Forem: Viljami Kuosmanen

We Made Coding Agents Actually Reliable By Fixing One Thing

Viljami Kuosmanen — Wed, 04 Feb 2026 15:55:45 +0000

Last week, Vercel published research showing that giving coding agents a compact index of your documentation dramatically outperforms letting them search for answers on demand. Their eval results: 100% task success rate with the map approach, versus only 79% when agents had to actively look things up.

Same agent, same tasks, different approach to context. The difference between working and not working.

The insight clicked immediately. If we could give Claude Code reliable access to this institutional knowledge without forcing it to decide when to look things up, it would fundamentally change how people work in our codebase.

So we built it.

The Context Problem

Coding agents have a token limit. A ceiling on how much information they can process at once. Think of it like working memory. You can't hand Claude Code your entire codebase and documentation library upfront. It's too much.

The traditional solution is skills: the coding agent decides when it needs information and actively looks it up. "I need to know about authentication, let me search for that." Sounds reasonable. In practice it creates three problems:

Decision paralysis - the agent has to decide when to look up docs, and it often guesses wrong
Async delay - every lookup is a round-trip, breaks flow
Sequencing conflicts - exploring code vs. consulting docs creates timing issues

Vercel's approach flips this: give Claude Code a compressed index of what documentation exists and where to find it, before it starts work. The index is small enough to fit in context every turn, so it always knows what's available. When it needs details, it reads the specific file directly. No decisions, no lookups, no conflicts.

Why Compression Matters

The key innovation is compression. A full documentation tree - all the folder structures, file names, categories - takes significant space. Too much to include in every conversation turn.

The compressed index uses a simple pipe-delimited format that shrinks this by ~80%:

[epilot Docs Index]|root: ./.epilot-docs|00-general:{tech-stack.md,business-context.md,ci-cd.md}|01-apis:{api-design.md,calling-apis.md}|02-epilot360-microfrontends:{env-vars.md,local-dev.md}|...

That single line tells the agent:

Where the docs live (.epilot-docs/)
What categories exist (00-general, 01-apis, etc.)
What files are in each category

It's a table of contents, not the full book. But it's enough. The agent sees the map, understands what's available, and pulls specific files when needed. The decision-making load disappears.

Why This Matters

The accuracy improvement is significant (100% vs 79% task success), but it's not the real story.

The real story is what happens when you lower the barrier to contribution. Proper context enables the person closest to the problem to fix it, regardless of job title. Your PM can fix bugs. Designers can adjust component behavior. Support engineers can patch data issues. You remove bottlenecks. Context and authority live in the same person.

Coding agents are the equalizer. But they're only as good as the context you give them.

Most companies will throw Claude Code at their codebase and wonder why results are inconsistent. The agent hallucinates patterns. Makes incorrect assumptions. Writes code that doesn't match conventions.

The difference is context. Structured, compressed, always-available context about how your codebase works.

How to Build Your Own

The pattern is straightforward. Here's what we did at epilot:

Curate agent-friendly documentation - organize your internal knowledge: conventions, APIs, architectural patterns, framework usage, code style
Structure by domain - group related docs (general, backend, frontend, infrastructure, etc.)
Use descriptive filenames - the agent sees filenames in the compressed index before opening files. api-design.md is better than guidelines.md. error-handling.md is better than errors.md. Make filenames searchable and specific.
Automate updates - pull live data where possible (OpenAPI specs, schema definitions, framework docs)
Generate compressed index - use a simple format (pipe-delimited works well) that reduces the doc tree by ~80%
Embed in agent context - add the index to your CLAUDE.md or AGENTS.md file (the context files Claude Code reads)

One thing worth highlighting: we don't just include our internal documentation. We also package docs for the frameworks and libraries we heavily use - single-spa, openapi-backend, openapi-client-axios, i18next, and Volt UI (our custom design system). When Claude Code needs to know how i18next pluralization works or how to register a single-spa parcel, it already has the answer. No hallucination, no outdated Stack Overflow posts, just accurate framework documentation.

What This Enables

We're already seeing daily usage across the team. Developers context-switch between services faster. Non-engineers contribute directly instead of filing tickets.

But the real potential is broader: if compressed context improves coding agents for technical documentation, why stop there?

Runbooks and incident response - on-call engineers with instant access to procedures
Customer domain knowledge - support teams with context on product behavior
Business logic - product decisions and their rationale, preserved and accessible

The pattern is the same: curate the knowledge, compress it, embed it in context, let the agent work.

The Constraints Are Disappearing

For decades, contributing to a codebase required deep technical knowledge. You needed to understand the language, the frameworks, the architectural patterns, the implicit conventions. The barrier was high.

Coding agents lower it. Claude Code, Cursor, and similar tools don't replace engineers. They make technical knowledge more accessible. With the right tooling and the right context, a PM can fix bugs. A designer can adjust styling logic. A support engineer can patch data issues.

The question isn't whether this is possible. It's how fast you can adapt.

Organizations which enable broader contribution will move faster than those that don't. The tools exist. The research is clear. What's missing is execution.

Start Simple

You don't need to document everything upfront. Start with the knowledge that causes the most friction:

Code style conventions - how you write TypeScript, naming patterns, file structure
Common patterns - how you handle authentication, API calls, error handling
Framework specifics - non-obvious usage of your frameworks and libraries
Internal APIs - if you have OpenAPI specs, even better

Create a simple doc structure:

docs/
  00-general/
    code-style.md
    tech-stack.md
  01-apis/
    api-design.md
    calling-apis.md
  02-backend/
    error-handling.md
    database-patterns.md

Generate the compressed index (pipe-delimited format, one line per directory). Add it to your CLAUDE.md or AGENTS.md file - the context files that Claude Code and other coding agents read on startup. Done.

The compressed index approach works! 🎉 Vercel's research proved it: 100% task success versus 79% without it. We've validated it internally. Now it's about whether you'll adopt it before your competitors do.

Dealing with Pushback to Product Engineering

Viljami Kuosmanen — Tue, 08 Apr 2025 13:34:33 +0000

I was recently confronted by a product engineer colleague here at epilot, frustrated with some of his non-engineer colleagues who didn't seem to buy into the idea of involving engineers in the product process from the start.

An all too familiar and frustrating situation for many of us.

You think of yourself as a proud product engineer wanting to solve meaningful problems but then get slapped with a detailed feature specification designed by a group of non-developers without your input. Now they expect you to go deliver their vision.

This is very much the reality in most product teams. Calling yourself a product engineer doesn’t automatically mean your voice will be welcomed. And that’s totally okay.

The Ideal vs. the Reality

In my past writings, I've painted the ideal: engineers who understand customer needs, question roadmaps, challenge designs, and contribute beyond code. But the reality is often much messier.

Some PMs and designers love working closely with engineers. Others are still adjusting to the idea. And that’s fair. The product engineer mindset definitely isn’t the norm.

Let’s not forget, PMs and designers are often under pressure too. It’s only natural they sometimes default to the most familiar and streamlined path. One that doesn’t always include engineers in the early stages.

And let’s be honest. Sometimes engineers haven’t yet built the trust or skills to contribute meaningfully.

The Friction Is Normal

The moment you step outside your lane, you shouldn’t be surprised when the reaction isn’t overwhelmingly supportive.

You will face skepticism.

You might be seen as overstepping.

You will encounter:

Requests to give estimates for implementing someone else's design
Roadmaps shared as top-down mandates
UI prototypes handed over "ready for dev", expecting pixel-perfect implementation
Pushback from asking too many questions

This is the price of wanting to do more than just execute. And if we’re honest, many of us haven’t always shown up in these conversations in a way that earns trust.

Is this a culture problem? Not necessarily. Most teams don’t have a rule against engineers joining product discussions. It’s just not the default. It’s less about policy and more about patterns. Changing those patterns takes trust, initiative, and persistence.

At the end of the day, it’s the product engineer’s job to show that product engineering actually works.

Earn the Trust

Calling yourself a product engineer isn’t a free pass. No one hands you a seat at the product table just because you want it. You must earn it. Show, don’t tell.

Do the homework. Know the names of your customers. Know the business domain.
Talk to your colleagues, not just other devs. Find opportunities to interact with customers.
Ask helpful questions that sharpen the team's product thinking.
Dive into data. Gather insights. Find new metrics and ways to collect useful signals.
Bring value to the table. Demonstrate you understand the customer problem by making thoughtful proposals that move the product forward.

You need to show up in demos, RFCs, testing sessions, and reviews. Not just in code commits.

Why Bother?

Because it’s worth it.

We do this for our own professional pride. To put great products into the hands of happy customers. And into our portfolios.

We don’t challenge product decisions because we want to take over the PM’s job or undermine the work of our teammates. We do it because we care about impact. Because we want our effort to count.

Because it hurts to pour weeks of your life into something that doesn't work, only to discover it failed because no engineer was involved in the concept phase.

Our Leverage

And here’s something to remember: We, as engineers, hold real leverage. We are the only ones who can actually turn product decisions into reality. No idea ships without us.

So while we may not always get a say by default, we do get a say in how we show up and how deeply we choose to care.

Use that leverage wisely. And proudly.

The Sauna Epiphany: How I Got Product Engineering Wrong

Viljami Kuosmanen — Sun, 13 Oct 2024 12:23:43 +0000

If you've seen my posts lately you've probably seen a lot of talk about Product Engineers: software engineers who talk in customer problems and take pride in the products they build, not only the code and technologies they use.

My core thesis is that with the rise of AI, the expectations for software engineers are being redefined, moving away from narrow programming roles: backend, frontend, C#, Java, React, etc. fast becoming obsolete due to AI tools like Cursor lowering the barrier of entry and unlocking productivity with any technology.

The days when you could coast on your ability to churn out code are over. If you're still clinging to the idea that you're safe just because you know how to write code in a widely used language or framework you're in for a rude awakening.

I believe this trend is likely to be the biggest industry shift in my lifetime, even surpassing the agile movement of the last 20 years. And I'm not the only one talking about it. (#1, #2, #3, #4, #5)

But I got it wrong.

Not entirely wrong, but I made it more complicated than it needed to be. In my quest to define what it means to be a product engineer, I overlooked the power of simplicity and ironically did not think enough about the customer: the ambitious software engineer thinking strategically about their careers.

I overengineered it

In my earlier attempts to help engineers break out from purely technical roles, I published an extensive checklist to help engineers think and ask questions as product engineers. It covered everything from understanding the user and the market to measuring success and staying ahead of industry trends.

Here's a taste of that checklist:

Great stuff, but let's be honest — no engineer is going to run through a 15-point checklist to make a product decision. Just seeing this long list of questions can be totally overwhelming, especially if you're used to being in a technical role.

If we expect engineers to actually start doing this stuff, the core idea needs to be simple and easy to remember.

The Wake-Up Call in the Sauna

As I'm writing this post, I'm enjoying a company working vacation in a nice hotel in Mallorca with my epilot colleagues. I was having a conversation in the hotel sauna with a principal engineer where something he said in passing suddenly clicked for me.

He was venting about some engineers on his team who seemed to be diving head first into coding without bothering to understand the problem. "It's so easy. Every engineer should be able to answer what problem they're solving, who the customer is, and what the impact of the work they're doing is."

That hit me like a splash of cold water in a 100°C Finnish sauna.

He was totally right.

Boiling It Down to 3 Essential Questions

To think like a product engineer, it's already enough to start with just three simple questions:

What's the problem?
For who?
Why is this important?

Let's dive a bit deeper into each one.

1. What's the problem?

Stop coding for a second. Do you really know what you're trying to solve? Not the ticket description in Jira, but the real-world issue. If you can't articulate the problem in simple plain English, you have no business writing a single line of code.

2. For who?

Identify your user and customer. Sometimes they're the same person; other times, they're not. Understanding who will use your product (and who will pay for it) is crucial. It shapes the way you approach the solution and helps you tailor the experience to meet their needs.

3. Why is this important?

As engineers there's never a shortage of things we could improve or implement. If solving the problem doesn't make a meaningful difference, why are you wasting your valuable time? We're not here to build features that nobody uses or cares about. Connect your work to something that actually matters and helps build your track record.

By anchoring your work in these three questions, you immediately move from code monkey to a high value product-minded engineer. Still a rare breed. You're not just implementing features someone else decided to build; you're elevating yourself to a position to influence product decisions and build smarter.

Leverage Your Team—They're There for a Reason

The biggest mistake engineers make is thinking they have to find all the answers themselves.

Most of us are lucky to work in a product team with other disciplines like UX researchers, designers, PMs and business stakeholders whose job is to help answer these questions.

By leaning on your team, you not only find better answers but also foster a more collaborative and innovative environment.

Product engineering is a team sport.

Taking Action: Start Asking the Right Questions Today

So, the next time you tackle a new topic, pause for a moment before diving into code. Ask yourself:

What's the problem?
For who?
Why is this important?

Write down your answers. If you don't know, reach out to your team and find out. This simple practice can transform your approach to work, leading to better decisions, more impactful solutions, and a greater sense of ownership.

Congratulations, you just became a Product Engineer!

Join the Conversation

I'd love to hear your thoughts on this simplified approach to product engineering. Have you tried asking these three questions in your work? What impact did it have? Share your experiences in the comments below.

Consider giving the GitHub repository a star if the product engineer role resonates with you!

Bullshit Tech Roles (satire)

Viljami Kuosmanen — Tue, 09 Jul 2024 23:50:55 +0000

In any sufficiently well-funded or otherwise successful tech company, a set of new roles will inevitably emerge—so crucial and revered by our industry that they've practically transcended the need to produce any tangible work. These roles specialize in the fine art of enabling others, in the hopes that, one day, the actual builders might be enabled enough to deliver products to customers.

I'm talking, of course, about the agile coaches and scrum masters, the architects and platform engineers; the growth hackers and data-driven researchers. These vanguards of innovation are vital to any team that wants to maintain a perpetual state of being maximally enabled, well-informed, and facilitated to perform their work.

Let's begin with the often underappreciated roles of agile coaches and scrum masters, the only people in a company actually certified™️ to set up the correct processes and tools to run an agile team. This, of course, is paramount for implementing agile—the popular software development philosophy that explicitly tells us to avoid focusing on processes and tools.

Agile coaches and scrum masters are professionals whose primary job is to ensure all work in teams is performed in sprints, with daily stand-up meetings to guarantee everyone is constantly enabled. This allows the velocity of a team to be meticulously measured in story points, which no one quite understands, but hey, at least we're definitely data-driven.

Surely, without their meticulous facilitation, teams would be left wandering aimlessly, unsure of how to break down their work into JIRA tickets. Their presence ensures that everyone stays aligned and that every team member is reminded of their blockers and sprint goals every single morning. Because nothing screams productivity like a daily stand-up where everyone takes turns saying, "no updates from my side."

Next, we have the architects. These exceptionally gifted, usually very senior engineers haven't touched any production code in the current decade. They are often found in their natural habitat, PowerPoint, where they sketch out their grand visions of future systems and envision platform rewrites for actual software engineers to implement.

Their role is to think so far ahead that their ideas become completely theoretical, creating software designs that are so advanced, they may never actually be implemented. But that's not the point. The point is to have a shared vision, to inspire, to draw boxes and lines that will one day guide the hands of actual engineers who might one day understand what they were trying to convey.

Then, there are the platform engineers. A demanding role with zero responsibility to deliver anything of value to customers. These unsung heroes dedicate their time to building and maintaining the internal tools and infrastructure that supposedly makes everyone else's job easier. Yet, their true skill lies in making things so complicated that only they can understand and manage them.

They manage infrastructure, design intricate CI/CD pipelines, common libraries and tools that add layers upon layers of necessary abstraction, which they insist must be adopted and standardized across all teams. The result? All teams having to learn and depend on custom in-house tools so complex that even the slightest changes require a detailed consultation with the platform team, ensuring their perpetual job security.

And then we have the growth hackers, the avant-garde marketers of the tech world. Their job is to come up with innovative ways to "hack" company growth, often resorting to questionable methods that straddle the line between clever marketing and outright deceit. They dive into data, running A/B tests, tweaking landing pages, and optimizing user funnels to squeeze out the tiniest incremental gains.

Of course, the real growth usually comes from the core product being genuinely useful, but why let that overshadow the need for an entire team devoted to marginal tweaks and vanity metrics?

Data-driven researchers are the prophets of the modern tech age. Armed with vast amounts of data and research, they uncover profound insights such as "users prefer faster load times" or "clearer buttons improve user engagement." Their work truly brings scientific rigor, managing to bring impressive-sounding numbers and great data visualizations to argue for any side of a decision, usually the one they already went with before doing any research.

These bullshit roles are the pillars upon which modern tech companies stand. They enable a culture where productivity is constantly measured, documented, and discussed, albeit often at the expense of actual productivity.

Without them, who would ensure that calendars get filled with back-to-back meetings so that everyone is too busy to notice no work is getting done? Who would write all the documents and Slack messages, ensuring no detail is left unshared or undiscussed? In doing so, these roles truly create a seamless, endless flow of communication, although one might argue that less talk and more doing might yield better results.

But where's the fun in that?

The Product Engineer Checklist

Viljami Kuosmanen — Tue, 04 Jun 2024 09:34:22 +0000

Download the PDF at productengineer.org

How to Think Like a Product Engineer

The following is a checklist of questions for product engineers:

1 Understand
- 1.1 Who's the user?
- 1.2 Who's the customer?
- 1.3 What's the market?
- 1.4 Ask Why
- 1.5 What do we already know?
2 Craft
- 2.1 Am I proud of what I'm building?
- 2.2 Does the product feel good?
- 2.3 How do I get there faster?
- 2.4 Teamwork
3 Growth
- 3.1 How do I measure the success of my work?
- 3.2 How do I maximise the impact of my work?
- 3.3 How do I stay ahead of the curve?
4 Product Vision
- 4.1 What’s our North Star?
- 4.2 How does my work impact the overall design of the product?
- 4.3 What's the ambition level?

1 Understand

1.1 Who's the user?

As a product engineer, your #1 goal is to create happy users. These are your fans! Always start with the user!

Your user is the person that primarily interacts with your product and whose experience your work will directly impact.

You may have multiple user groups. People who may have varying reasons to use your product or might interact with it in different ways from different angles, maybe on different kinds of devices.

You should understand how to help these users. What drives them to use your product? What delights them? What are their pains?

As a product engineer you should demand and support your team find answers to these questions before jumping into writing code.

1.2 Who's the customer?

Yes, this is a different question to "Who’s the user?". The customer is whoever pays for your product, not always who uses it.

You should know what makes your product valuable to your customers to make better decisions on what to invest your time.

Hint: If you're in B2B the answer always has to do with helping your customers save money or make more money. Understanding your customers' core business is key to understanding why they would pay for your product.

Most importantly, you want to make whoever is paying for your product look good. After all, they're the ones taking a risk by picking your product. You ALWAYS want to reward them for that.

1.3 What's the market?

Zooming out, let's take a look at the wider market landscape. Who are the potential customers we haven’t captured yet? Why would a customer pick a competitor’s product vs. mine? Are we leaving opportunities on the table?

What can I learn from similar products in the market? What are our USPs? How do I create a competitive advantage against competition?

Are there rules to the market? Are there regulations or industry standards I need to know about? Does my product have to look or feel a certain way to be taken seriously?

Knowing the market and regularly bechmarking yourself against other players helps become aware of your strengths and weaknesses and give ideas for where to invest strategically in your own product.

1.4 Ask Why

This is a bit of a product thinking cliche but still holds true: always pays to ask “why” a few times to uncover root causes of problems and underlying motivations.

Asking why can be helpful in almost any situation to build understanding in your team. Some examples:

Why should we invest into building this feature?
Why are customers asking for this feature?
Why is this a pain for our users?
Why do users give us that feedback?
Why now?

1.5 What do we already know?

It’s smart to build on what’s already known rather than always starting from scratch.

Always leverage the existing knowledge and experience of peers and leaders: founders, product leadership, designers, other product engineers, etc.

Examine the status quo to see how a problem is currently solved. Look at ideas, feedback, metrics, KPIs already collected in the past.

Do we have users already doing something like this? What are their existing workflows? How could we improve their experience?

Am I duplicating or potentially deprecating some functionality that already exists? Could this be achieved by extending or leveraging existing features? How are competitors solving this?

2 Craft

2.1 Am I proud of what I'm building?

Your track record as a product engineer is the products and features you've delivered. Your last feature represents your professional competence level. No excuses.

Ask yourself what quality standard do I want to set for the work I put out there?

Can I be proud of the product I worked on? Is my work well tested and polished? Did I cut corners where I shouldn't have?

As a highly paid professional engineer your craft is to produce high quality software which includes avoiding the creation of technical debt. Never ask for permission to improve quality!

What makes a great product engineer stand out from the average software engineer is an intense sense of professional pride in their work and product.

2.2 Does the product feel good?

This may be a slightly controversial take, but I believe a surprisingly large part of building great products is about developing a good taste for it.

Simply knowing the difference between great vs. average and not settling for just ”ok” helps tremendously in making good decisions as a product engineer.

Does the product feel smooth and consistent? Is it intuitive, simple, and familiar to the user? Or does it feel cheap and janky?

Note that this doesn’t only concern the visual aspects of your product. Just slapping a fancy UI design on a shaky foundation doesn’t create a great experience.

Simple. Elegant. Clean. This is what we're after as product engineers.

2.3 How do I get there faster?

The pace of innovation especially in software is so rapid that in order to be competitive you must deliver fast, early and often.

Too slow and your customers will lose trust in you while your competitors overtake you.

What many get wrong about agile and building products is optimizing for predictability with estimates and roadmaps. Rather, what you really should care about is visible and continuous progress towards product goals.

The goal of estimates should not be to try to be as accurate as possible but rather to set ambitious and yet achievable goals for yourself.

The question is not how long you think it will take to build, but how long should it take? What’s an acceptable amount of time and effort I should invest on this?

What’s the rollout strategy? How do I get this into customers' hands as soon as possible? What’s the MVP?

2.4 Teamwork

Building products is a team sport.

You are absolutely not expected to work alone and do everything yourself from writing code to doing user research.

Am I effectively communicating with my team? Am I leveraging my team members' strengths? Are we celebrating our successes?

Great teamwork results in great products. Invest in your team, and you will see the dividends in your product's success.

3 Growth

3.1 How do I measure the success of my work?

Our work as product engineers is not just about building and shipping feature after feature.

We make educated guesses about the most valuable thing to work on, so we should also be able to answer the question: What’s the impact of my work?

How many customers did we talk to to validate our progress? Are they coming back? How much money does it generate?

Use analytics tools and user feedback to help you understand what’s working and what’s not. Talk to real users to get qualitative insights about the product.

Most importantly, make sure to share your outcomes openly and transparently. What did I ship in the last few months? What were the results?

3.2 How do I maximise the impact of my work?

The most important question you should regularly ask yourself is: am I focused on the right thing?

Being a good engineer is finding the biggest bottlenecks that hold us back and figuring out how to solve them. You should prioritise your efforts ruthlessly.

Building products is a team sport. You should actively communicate what you’re working on and why, so that others can help you and keep you accountable. Demo progress frequently and actively seek feedback.

Don’t fear taking risks. Being bold and taking the lead on delivering new and innovative features you believe in can lead to big wins.

Ask yourself: How can I align my work with our company goals? How does my work directly benefit our users? Is there a way I can communicate my work better to others to get better feedback?

3.3 How do I stay ahead of the curve?

Stay updated on market trends. Attend conferences, read up, follow experts.

Make sure to research and benchmark competitor products, or other similar products whenever possible.

Hold frequent retrospectives and brainstorming sessions. Experiment with new ideas. Encourage creativity in your team.

What are the latest trends in my industry? How can I encourage innovation within my team? Are we taking enough time to learn from our successes & failures?

4 Product Vision

4.1 What’s our North Star?

To align your work in the context of the broader product vision, you should deeply care about what others in the company are doing and saying, making sure you fully understand and are committed to the overall product strategy. Asking “why” is crucial here.

What are our current product goals? What’s our growth strategy? Where do we want to be in the next 2-5 years?

When presenting your work it always helps to put it in the context of how it pushes us forward in the big picture. How does my work help us reach our North Star?

4.2 How does my work impact the overall design of the product?

Understanding the existing software’s design and architecture decisions helps make sure that your work fits well within the larger product design. Always consider how your work influences and is influenced by other components.

When adding new functionality, you should ask:

Could this be achieved by extending or leveraging existing core features?
Does my design follow a consistent style to the rest of the product?
Am I adding complexity or reducing it?

Simplicity is worth fighting for.

4.3 What's the ambition level?

It’s a good idea to define the ambition level when setting off to build something new. Are you aiming for an incremental improvement or a revolutionary new functionality?

Is this feature a USP or a basic expectation? What’s the ROI on building this feature? Does it make sense to spend the effort building this ourselves, or could we use an off the shelf library or service?

Your time is extremely valuable. Think like an owner: Would you invest your salary to build the feature, or rather on something else?

Product Engineer Mindset

If you enjoyed this checklist, you might also like the Product Engineer Manifesto on GitHub.

Consider giving the repository a star if the Product Engineering philosophy resonates with you!

What is a Product Engineer?

Viljami Kuosmanen — Thu, 29 Feb 2024 08:04:56 +0000

Read the full Product Engineer Manifesto at productengineer.org

Not Just Coders, but Builders

Picture this: engineers who don't just speak in code but in product design and customer problems.

Product Engineers are a special breed of engineers that don't just see themselves as coders or developers but as builders who deeply care about the products they build.

They're driven by a professional pride and desire to build great products, transcending the traditional developer role to become genuine drivers who go on to put their names behind great products they themselves played a key role in shaping.

The Counter to Hyperspecialisation

The tech world has seen for a long time a trend towards hyperspecialisation, with engineering roles becoming increasingly narrow.

We have frontend engineers, backend engineers, iOS engineers, DevOps engineers. We have developers defining their careers with a specific language: JavaScript engineers, Swift engineers, C# engineers, Python engineers. We even have developers that seemingly dedicate their entire careers to a single framework or tool: React Engineers, .NET engineers, Unity Engineers, Ruby on Rails engineers.

Feels like it's only a matter of time until we see job postings seeking for-loop engineers and variable naming engineers.

Jokes aside, specialisation isn't totally without its merits as it has allowed engineers to build deep expertise in specific technologies, a key component in building quality software. However, it also lead to silos where collaboration and broader product understanding became a nice-to-have, and often not even an expectation or focus for engineering roles.

Product Engineers stand as a counter-movement to this trend. They embody a holistic approach to engineering, where understanding the entire product and context around it is just as important as the technical skills required to build it. This broad perspective enables them to bridge gaps between different technical domains and ensure that the product serves its customers effectively.

AI: The New Playground

The rise of artificial intelligence (AI) tools in software development is rapidly setting new expectations for engineers. With AI tools becoming more sophisticated and capable, engineers are now expected to leverage these tools to enhance their work, not just in terms of speed and efficiency but also in terms of broadening their area of responsibilities.

Engineers in a post- ChatGPT and GitHub Copilot world are now expected to be able to work in more languages, incorporate more tools and libraries, and simply be able to deliver more, using a broader range of technologies and disciplines in their daily work.

It's probably not a great time to be the platform engineer whose job is to set up Jenkins pipelines for the product teams, when ChatGPT can easily help teams do it themselves.

For Product Engineers however this has unlocked a higher level of abstraction. With AI tools handling more of the routine and specialised tasks, engineers can allocate more time to creative problem-solving, ideation, and exploring new ways to meet customer needs.

This new level of abstraction allows Product Engineers to concentrate on product strategy, user experience, and overall system architecture without being bogged down by the intricacies of individual technologies.

The incorporation of AI into software development encourages a broader perspective, where the choice of technology becomes a means to an end, rather than an end in itself.

While AI didn't create the Product Engineer role (examples have been around for much longer); AI has made becoming a true Product Engineer much more accessible as a career path for software engineers.

Combining Product Thinking and Technical Execution

Product thinking involves understanding the user's needs, the market demands, and the business goals that drive a product's development. It's about seeing beyond the immediate task to grasp how each piece fits into the broader puzzle of the user experience.

Product Engineers ask the crucial questions: "How does this feature add value to the user? What impact will this improvement have on the product's overall vision? How can we measure the success of our work?"

They don't just build features based on specifications; they contribute to the design and roadmap of the product through a robust understanding of the customer's needs and business strategy.

This approach requires a balance of skills: the ability to dive deep into coding and system architecture, while also keeping an eye on the product roadmap and customer feedback.

By combining this mindset with quality technical execution, Product Engineers can uniquely ensure the product is headed to the right direction both technically and strategically, ensuring that their work directly contributes to the product's long term success.

The Manifesto: A call to Product Engineers

I've published a Product Engineer Manifesto on GitHub to more formally define the Product Engineer mindset.

Consider giving the repository a star if Product Engineering philosophy resonates with you!

After 3 years: Bets I made as Head of Engineering rewriting the epilot SaaS – worth it?

Viljami Kuosmanen — Thu, 15 Feb 2024 21:56:48 +0000

In this post, we take a look back at some of the most influential decisions I implemented early on in our SaaS rewrite journey at epilot and evaluate how well each one has played out after 3 years.

Skip ahead or read on until the conclusion for a full verdict of whether rewriting our app was a good idea to begin with.

Doubling down on AWS & serverless

Good bet 👍

To be fair, AWS was already the de-facto cloud platform used by the epilot team before I joined.

My contribution was to convince the team to go all in on a serverless architecture with an ambitious strategy to rewrite and replace our proprietary Java-based 3rd party monolith running on EC2, piece by piece with functionalities leveraging serverless AWS offering.

After 3 years, there are no regrets to choosing to be fully vendor-locked with AWS for the speed, scalability and minimal operations overhead it provides.

Microfrontends with single-spa

Good bet 👍

The decision early on to take a microservices approach in our frontend was made out of purely practical reasons.

Frankly our small team who lead the work on the new platform didn't have the time, the confidence, nor the desire to make long term decisions on the frontend tech stack, tooling and important design decisions on behalf of all our other teams, present and future.

I'm extremely happy we went down this path instead of imposing one framework and toolset for everyone. The single-spa approach lets us improve our frontend game each time a new microfrontend gets added to our application, which by now has happened more than 30 times! We've even ended up replacing entire MFEs (microfrontends) a few times, which isn't as bad as it sounds when the codebases are kept relatively small.

Notably the microfrontend approach also helped us support the transition from the legacy product to the new one, as we were able to treat the legacy app as just one of our many (micro)frontends, and slowly replace functionalities one by one by shipping new microfrontends.

Using React without a framework

Bad bet 💸

I know, "skill issue", right? But honestly I've found that even the really senior (read: expensive) React engineer hires tend to struggle with performance and end up wasting a lot of time on fixing major performance issues with React applications when they start to do anything meaningful.

Starting over, I would probably spend a bit more time choosing a React framework that works well in a microfrontend environment rather than encouraging all teams to start from scratch, and establishing no-framework as the implicit default.

Controversial: I might not even pick React for most MFEs anymore. We already use Svelte in parts of our app, and turns out Sveltekit works super well in a single-page-app environment like ours!

RFCs

Good bet 👍

The practice of writing RFCs to collaborate on software design and async decision making was one of the first and best things I introduced joining epilot.

To this day we use RFCs as the main way to share ideas and be transparent about the work that our engineers do. I'd consider this one a very good bet and would absolutely recommend a culture of writing RFCs for all product organizations.

API First

Good bet 👍

We give a ton of freedom to engineers to pick whatever languages, tools and techniques they see fit, but the one thing I've always demanded from our teams has been to design using machine-readable API contracts, such as OpenAPI or GraphQL.

I would say this is internally the most important way we collaborate on our software's design, using shared types and automated tests to ensure our implementations follow the common API contracts. This approach has radically reduced bugs and allowed our engineers to work together efficiently leveraging platform features built and maintained by other teams.

On top of that, being API First and publicly building and sharing our API contracts has allowed our customers and partners to effortlessly build on our product, and integrate it deeply with their own IT landscape using our API & SDK.

Continuous production deploys from `main`

Bad bet 💸

Turns out our customers simply did not tolerate the frequent changes and inevitable rollbacks of this approach.

We technically still ship continuously from main to our production environment, with an average of 150 production deployments every week, (with a lot of tests!), but now ship a monthly stable snapshot release to paying customers, and heavily utilise feature flags to roll out changes.

Not using feature flags from the beginning

Bad bet 💸

We did use GitLab feature flags on launch, but they were only used to turn features on/off between stages (dev, staging, production).

Turns out this wasn't enough. Our teams really needed better tools to give us fine-grained control over rolling out features to specific customers.

Going back, I wish we had started earlier with something like LaunchDarkly.

Hiring product-focused senior engineers remotely

Good bet 👍

This is a decision I'm very proud of. Keeping an almost unreasonably high bar to getting hired and seeking out individuals who care about the product, our customers and our business has allowed us to build and retain a motivated team with zero managers and architects.

Making engineer hiring remote-first allowed us to tap into an international pool of senior talent resulting in a team of more than 30 exceptional and motivated hands-on Product Engineers from more than 10 countries working for us, and growing... ❤️

In-house DevOps / platform team

Bad bet 💸

We tried to establish a platform team on two separate occasions but both times the team struggled to produce value for our teams. The DevOps engineer hired to lead the second platform team ended up leaving us for a bigger company after a few months.

Looking back, part of the reason for this was likely a combination of leveraging managed/serverless services that don't require much in-house automation work, and hiring senior engineers who can automate things for themselves when needed.

We've recently started experimenting working with a 3rd party DevOps consultancy to join us on a project-basis to help us with non-product related engineering tasks such as optimising our CI/CD pipelines.

Regardless, we don't anymore see any need to hire and establish an in-house DevOps team, or get into Platform Engineering of any sort in the foreseeable future.

Weekly Demos

Good bet 👍

The all-hands weekly demo is still easily my favourite recurring event at epilot. It's like having a company-wide celebration of shipping things, every week!

The weekly demo brings not only a positive buzz around all the cool stuff being built, but also the expectation that engineers personally showcase concrete customer value created in their teams every week. It keeps us accountable for consistent delivery and maintains a rapid feedback loop with the rest of the company.

Open-Source Engineering Principles

Good bet 👍

Transparently laying out our Engineering Principles in a public GitHub repo helped set clear expectations for our culture internally, and perhaps even more importantly for engineer candidates, many of whom have testified making the decision to apply at epilot because of our principles resonating with their own beliefs.

Hasura

Good bet 👍

One of the early key bets was the decision to use Hasura to interface with our legacy database to break up the monolith into microservices.

Event triggers listening on DB changes and pushing messages to SQS while using the GraphQL API to implement 2-way data sync with the legacy app turned out to be a great and reliable event-driven pattern for breaking up the monolith step by step.

Since shutting down the legacy product last year, Hasura was decommissioned together with the legacy Postgres database, but it truly served its purpose well during the migration to serverless. 🫡

DynamoDB as our go-to database

Good bet 👍

We haven’t had any fundamental issues with DynamoDB single table design in our backend microservices.

Of course it’s not suitable for every use case, and we have turned to Aurora serverless, Elasticsearch, and even InfluxDB in a few special cases, such as for search and analytical needs. But for 80% of the time, DynamoDB is fast, convenient and extremely worry-free as a default application database.

AWS OpenSearch service

Bad bet 💸

We started out on Amazon OpenSearch Service (still called Elasticsearch service back then) but were struck with a pretty bad incident where our domain became completely non-responsive in our development environment and the only thing we could do was to contact AWS support to resolve the issue. This was totally unacceptable for us as we heavily rely on Elasticsearch for our main functionalities.

Elastic Cloud ended up being far more reliable and manageable for us in the end. Having a self-service reboot button on the interface makes all the difference!

However with both providers rolling out their serverless solutions, which would be extremely attractive to us, we're keeping our eyes open and are open to experimenting again with AWS OpenSearch offering.

Datadog for observability & monitoring

Good bet 👍

Datadog is one of those tools that once you get used to it, you wonder how you ever lived without it.

The value we get from Datadog's UI for logs, traces, monitors and dashboards is insane, and absolutely worth every penny.

We use Datadog RUM for browser & UX monitoring. The RUM sessions and logs are extremely helpful for tracking down frontend crashes and bugs, as well as visually understanding user behaviours.

On top of that, we use pipeline observability to analyse CI pipelines, custom metrics to track business & application KPIs, and cloud security management for compliance reporting and threat detection.

Datadog for incident management & post-mortems

Good bet 👍

Datadog Incident Management has been absolutely vital in making sure we take the right steps to analyse and learn from our incidents. We use it during incidents for coordination and communication, as well as after the fact for post-mortem and analysis of past incidents and trends.

Datadog for browser tests

Bad bet 💸

Datadog synthetic browser tests were great to get started fast but quickly became slow, expensive, hard to expand and simply not good enough for our team.

We ended up switching over to Playwright.

Redshift Serverless

Bad bet 💸

Quite slow. Very expensive. At least for our use case of generating dashboards with Spectrum querying mostly parquet files from S3.

Our team is currently investigating replacing Redshift with Clickhouse.

Microservices

It's complicated 😎

At this point I've stopped referring to our product as having a microservices architecture. I don't find it a very helpful abstraction to understand a large, event-driven, interconnected product that's mostly FaaS.

Yes, we still develop and deploy our services as independent domain-specific modules with well-defined APIs.

In reality however our application is used by the customer as one big product with all the pieces needing to connect and work together to provide a service.

Thus, we must also test and ship the product as one. (Really, we have one large shared e2e test project that blocks all pipelines. When we're shipping our monthly stable release, we take a snapshot of all our bundles and ship those in one atomic import map file.)

I simply find it more useful to think of our product as one monolith, made up of a large number of independent infrastructure & FaaS modules with APIs.

Conclusion: Was the rewrite a good idea?

If you ask me now I would never recommend any SaaS company take the decision to fully rewrite their software. It's generally just a pretty dumb and painful thing to do. 😅

For epilot sadly we were left with no other choice.

The legacy version of our SaaS was built on top of a proprietary 3rd-party low-code platform Axon Ivy, which by no means was designed to run a multi-tenant SaaS, but that the external epilot dev team from Vietnam (affiliated with Axon) had managed to wrangle to work.

The fact that epilot was able to build a successful and fast-growing business on top of what's essentially a hacked low-code business process automation tool, is the ultimate proof to me that the tech stack really doesn't determine a company's success!

But we were seriously starting to hit the limits of our software, especially when it came to scalability and speed of development. We didn't even have access to the source code so that we could take over the development of the core application. All we could do was upgrade to new versions provided by the vendor and continue to hack the software to make it do more things it wasn't designed for.

Simply put, it wasn't sustainable for a fast growing software company. No developer wants to work on software like this. Especially not the talented and ambitious ones we wanted to work with.

It took us nearly 3 years, but in the end we managed to reach feature parity, replace, migrate and shut down the legacy app, with very little churn, growing year-over-year and last year doubling our MRR while migrating the last customers over to the new platform.

Feels good to say we're only getting started with our new 360 platform.

Looking back at the last 3 years, while I would never recommend anyone pursue a rewrite for a running SaaS business, while simultaneously trying to build a healthy, ambitious engineering team and culture, I'm convinced that for epilot this was the (only) way to go, and we're much better for it.

In the end we at epilot overwhelmingly consider this a…

Good bet 👍

What's next?

To keep things short in this post, I'll publish a separate Part 2 where I share our latest bets focused on building epilot into the next SaaS Unicorn. 🦄

Mandatory recruitment disclaimer: If this post made you at all curious, please check out our docs and check our open positions for more!

We’re hiring @ epilot

Comparing REST, GraphQL & tRPC

Viljami Kuosmanen — Tue, 10 Oct 2023 10:59:54 +0000

As an author and maintainer of open source libraries to work with REST APIs, and as someone making a living building an API First SaaS product with both REST and GraphQL APIs, I sometimes encounter strange attitudes towards different API styles.

"REST is outdated" "GraphQL is only good for mobile dev" "tRPC is too new for production" "No one likes to work with swagger"

It might be tempting to pit these technologies against each other, but the reality is, these paradigms are not direct competitors. Instead, each has its own unique strengths and use-cases, crafted for specific contexts and developer needs.

I was reminded of this fact recently when writing comparisons between my own open source REST tooling vs. GraphQL and tRPC:

// Detect dark theme var iframe = document.getElementById('tweet-1711285703840084201-562'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1711285703840084201&theme=dark" }

Let's take a deeper look at each technology and its strengths:

GraphQL

GraphQL is a query language for APIs developed by Facebook. It gives API clients full control over the data they query, making it extremely flexible and efficient for client-centric use cases.

GraphQL APIs define a strongly typed schema for the data and mutations they support which makes them discoverable and intuitive to develop against.

Teams building with GraphQL naturally take the API First (or Schema First) approach where the API contract is treated as a first class citizen in the software design instead of treating it as merely documentation.

When developing an app with a GraphQL API, it's common to use the GraphQL schema as the single source of truth for data models used within the application, and source types from it used end to end across the codebase.

GraphQL APIs are often used as gateways, merging multiple downstream APIs into a single, easy to use interface for a great application development experience. I would consider this one of GraphQL's killer apps.

Facebook's excellent Apollo GraphQL is definitely a best-in-class example of great API development experience.

REST

REST has been around for a while now. It's still probably the most widely adopted HTTP API design paradigm perhaps due to its resource-based implementation simplicity and the ability to leverage standard HTTP features directly.

With widely adopted open standards to define REST APIs, like OpenAPI specification (previously known as Swagger), REST can be used much like GraphQL with a single source of truth for the API contract.

Full disclosure: my own open source work in the last few years has revolved around the mission of bringing a GraphQL-like developer experience and type safety with OpenAPI for REST.

While REST APIs don't generally provide the same level of control to clients as GraphQL, sometimes this could be seen as a benefit especially in scenarios where strict control over data access and operations is crucial.

Widespread knowledge around REST contribute to its choice among organizations looking for a tried-and-tested approach to building APIs.

tRPC

While RPCs are by no means a new idea, tRPC is seen as the new kid on the API block. Much loved by developers who just want to build fast and do it with type safety.

Designed for full-stack Typescript applications, tRPC allows direct sharing of types between both the client and server, without relying on code generation.

Unlike GraphQL and REST, tRPC doesn't expose a standard machine-readable API schema to be consumed by clients, instead taking a more straightforward approach of exposing endpoints or procedures, essentially "just functions" invoked by the client to the server.

While the lightweight tRPC approach is optimal for teams just looking to build full stack applications, teams looking to build robust, re-usable APIs may better be served by the API design first approach of GraphQL or OpenAPI.

(I wonder if tRPC will eventually get its own YAML specification language)

Conclusion

In the landscape of software dev, it's easy to get caught up in the rhetoric of one technology being superior to another. But as we dive deeper into the nuances of GraphQL, REST, and tRPC, it becomes evident that each serves its unique purpose and has been created to solve specific problems in API development.

The right choice of an API paradigm is definitely not a one-size-fits-all solution.

It's essential to remember that the success of an API doesn't necessarily depend on the technology backing it, but more on its clarity, usability, reliability, and the developer experience it offers.

Technologies change, but the principles of good API design and the goal to write delightful, working software remains constant.

If you're interested in my Open Source work around API tooling, check out openapistack.co

How we do engineering @ epilot

Viljami Kuosmanen — Sun, 22 Jan 2023 15:35:57 +0000

Ever since open-sourcing our core Engineering Principles back in January 2021, I regularly get asked by candidates how and if the principles really work in practice at epilot.

In this post, I'll attempt to give a glimpse of concrete tools and practices we use in our product team to give an idea of how we apply our principles and what the engineering culture really looks like from the inside.

Team Structure

At time of writing, the epilot engineering team consists of 25 full-time remote software engineers organized into 6 squads, each with its own Product Manager and a shared design team of 4 talented product & UX designers.

Each squad is led by an engineer team lead, principally responsible for the team's delivery and engineer wellbeing.

The product leadership team is responsible for communicating our product strategy and maintains a Now-Next-Later roadmap, but ultimately it's the squads that make all delivery decisions.

Squads are formed around specific business domains within the epilot 360 platform, architected around Domain Driven Design and micro-ish services both in the backend and frontend.

Show, Don't Tell

We value real concrete working software, not roadmaps and fancy presentations.

Engineers showcase features in a weekly all-hands Product Demo session: a company-wide celebration and ruthless feedback session. 🎉

Features being demoed are deployed to production but are usually hidden behind a Feature Toggle, giving demo participants a chance to test and give feedback on features before release.

Most squads use a Kanban-style flow with a weekly planning cycle, but opt to showcase their progress every 2 weeks, with some squads hosting extra demos called "Open Houses" on off-weeks.

The engineering team also hosts an internal bi-weekly "Tech Exchange" to wind down with a beer, but also present and demo cool technical things.

API-first & serverless

The epilot backend is built as serverless microservices written mostly in Typescript, leveraging services such as AWS Lambda, Step Functions, EventBridge, API Gateway and AppSync.

DynamoDB is the most popular database solution enhanced by managed data stores such as Elasticsearch Cloud, Aurora Serverless and Redshift Serverless.

Our backend is pretty event-driven, with heavy use of EventBridge, SQS and Step Functions to implement asynchronous business logic.

Our frontend application consists of microfrontends written in React and Svelte orchestrated by the single-spa framework.

The epilot frontend is entirely built on the same APIs we expose to our customers and technical partners via our open-source SDK and public documentation.

We like open standards, so our APIs are defined using common machine-readable specs like GraphQL and OpenAPI.

We find the domain-driven architecture approach an efficient way to organize our teams to act independently while collaborating on software design with an API-first approach.

Organizing our codebase with microservices helps our fully remote engineering team focus on delivery and minimize bikeshedding over non-critical engineering decisions.

Freedom and Responsibility

Ok, I freely admit we stole this one from the Netflix culture deck. Shoulders of giants and so on. ♥

At epilot, every hire is trusted and treated as an adult from day one. We only hire smart people who we can trust and aren't afraid to terminate a relationship if that trust is broken.

We expect engineers to operate by principles and think strategically to benefit our customers and our business. (Max-the-MRR)

We intentionally make it very difficult to create company-wide policies to control which tools to use, what employees should be allowed to do and how they should organize their daily work.

It should be hard to introduce new processes but easy to get rid of existing bad ones.

The exception to this rule is security.

We encourage team members to challenge the status quo with RFCs proposing ways to improve the product and our ways of working.

RFCs - Solutions Over Problems

Among other things, RFCs written by engineers have so far led us to adopt continuous deployment, harden security, establish new teams, and even rewrite the entire product in a different tech stack.

These are often simple documents that start out by just describing a problem and expressing a wish or an idea to solve it.

RFCs at epilot have no official approval process. We expect the author to champion their proposal by garnering relevant feedback from peers and stakeholders, and then make an informed decision on how to proceed.

In most cases, the RFC author leads by example and implements the proposal themselves, usually just within her squad at first.

Good ideas spread and are adopted through all teams, not by mandate from above, but by actually improving the quality of our work.

Edit: A total of 192 RFCs have been published in our Confluence since they were introduced 2 years ago. 🔥

Every Week is Quality Week

The name of this principle came about after a group of engineers expressed concern about technical debt in their project due to pressure to deliver against deadlines.

At the time, it was already an established practice to fix bugs immediately and treat all types of bugs as critical, but many engineers still felt that they couldn't go against the wishes of product managers to follow the practice.

It was proposed to have a "quality week" at the end of every quarter to give engineers time to focus on quality topics.

As the quarter came to a close, we announced the quality week with instructions to all squads to focus solely on bugs, technical debt and other quality improvements they felt were important.

During the quality week, our engineers were extremely motivated to fix long-time issues that had affected our customers.

We made real, significant progress that week and everyone could see it.

On Monday following the first quality week, we opened this PR, making an announcement that every week from now on would be quality week:

It worked. We're still starting a new quality week every Monday.

No-Ops & Continuous Deployment

At epilot you will find a distinct lack of epilot-specific shared tooling. We actively avoid building our own tools and standardizing configurations and instead encourage using open-source tools like AWS CDK, lambda-powertools and single-spa to build our product.

We don't have dedicated DevOps engineers or a platform team to build custom tools and pipelines or run k8s clusters.

We expect squads to independently automate testing, deployment and monitoring of their features to minimize the need for manual operational tasks.

Engineers really get to pick any tools they like to do their job, as long as they weigh tech decisions against our principles.

I sometimes get engineers pushing me to leverage my position as head of department to standardize tech choices across the entire engineering team, but so far the approach of giving teams radical freedom has yielded organic adoption of best practices and tools where needed anyways.

Without extra push, all teams have converged to using the same GitLab CI templates, connecting up with Datadog to continuously deploy code with synthetic browser tests integrated into the pipeline to make sure we catch problems before they get to production.

The whole team also uses Datadog (another RFC) as the central observability solution to know what's happening in our platform at all times.

When stuff goes wrong, engineers owning the feature jump on a Slack huddle using Datadog's incident management feature to coordinate the response, cleanup and write a post-mortem.

Why principles matter

When I joined epilot, I made it my personal mission to build an engineering culture that I would personally want to join and be part of.

To me, the engineering principles are like a founding document; a source code or DNA of how to build a great engineering team.

I'm proud of how far we've come, but there's still lots to learn and improve as we grow and mature as a product team.

I fully expect that in 2 years, writing this blog post would look very different as the industry moves forward with new tools and practices. I hope that these principles will stick.

If the topics in this post resonated with you, make sure to check out our dev docs and see our open positions!

We're hiring @ epilot!

First year as Head of Engineering @epilot - Killing the Legacy Monolith

Viljami Kuosmanen — Fri, 17 Jun 2022 08:34:45 +0000

This post is a personal account of my experience at epilot.cloud and our journey rebuilding our legacy Java monolith SaaS from the ground up with AWS, serverless and microfrontends, mostly during 2021.

The text is written from the perspective of engineering and focuses on the challenges I faced taking on leadership of our growing team of 25+ engineers.

I fully acknowledge this is not the complete picture of epilot's growth during this time as we have many other amazing departments and individuals working to make epilot the next tech unicorn success story.

Full disclosure, I love this company and am heavily invested in the product's success after my first year and a half. 💙

(I'm the one dressed up as the doctor for our company's annual Cologne carnival celebration)

Beginnings

I joined the Cologne-based startup epilot in October 2020 as a cloud engineer to head a new platform team and aid the product team transition to serverless AWS.

The task felt right in my wheelhouse.

By that time I already had three real serverless products under my belt, had experience migrating monoliths to microservices, and had acquired more than 8 years of professional experience in total as a developer and team lead working mostly with AWS and other cloud platforms.

I knew startups. I had experience with the business domain. I knew the technology well.

This was my time to shine.

Build better software, faster.

As our CTO Szilard Toth put it in our first discussions, my job description was to help us "build better software, faster".

I took that mission statement very seriously.

Coming off a 4-year stint at Futurice, a lean service design focused consultancy, my first instinct was to make a current state analysis of my new company - epilot.

To get my bearings, I joined one of the engineering teams and got busy.

I learned about the challenges our teams were facing and got a deeper view into our legacy product, which had already been in active development for more than 3 years.

We were already doing a lot of things right: All three teams at the time were generally following agile practices, engineers were eager to discuss and learn their new tools, many had even taken the time to get AWS certified.

But there were certainly some big concerns.

At the end of my first full month, I was ready to present my learnings to the company.

First insights

I published an internal document titled "epilot Tech Review" detailing my assessment of the current state of epilot's engineering.

These were the main findings:

The decision made (before my time) to pick AWS, serverless and React as our foundation was a good one for what the team wanted to achieve: Low operating costs and high development speed. This was especially great for hiring due to many candidates being attracted to these modern technologies.
Our teams lacked experience working with these technologies. Progress was slow and unfortunate decisions were made with the very first small scale serverless projects.
Alarmingly, the engineering team had no viable plan or strategy to migrate the legacy product to the new serverless architecture. Everyone was excited about the new tech but we clearly had no clue how to actually build our product with it.

I emphasised in the document how important it was for epilot's success to move away from the legacy technology stack, a proprietary monolithic Java framework which had already proven to be a poor fit for our use case, difficult to run and extremely slow to develop new features on.

Our ability to deliver as an engineering team was looking pretty grim. The company had already invested a full year into going serverless with not much to show for it.

To make matters worse, some of our key engineers had just announced they were leaving the company.

In spite of our woes on the engineering side, epilot had a rare advantage:

A proven product-market fit – Even with our product leaving much to be desired, our paying customers were becoming fans!

Yes, they were sometimes frustrated with the quality of the software and our inability to deliver features they were asking for, but they were totally bought into the vision of what epilot could be and we were clearly delivering value for them.

I think this was thanks to our visionary leadership with deep industry knowledge knowing which problems to solve along with a great sales team and a MacGyver-like customer success team working around the limitations of our software.

We had the right idea, just not a great technical execution yet.

Luckily I knew I could make a difference when it comes to execution.

RFCs and API first

I went on to establish the platform team I was hired for. We started small with myself and an SRE who had been at epilot a year before me.

The first things the team tackled were improvements to developer workflows such as enabling an API Design First approach for our serverless APIs and building common CI/CD pipelines for the tools our teams were using.

I introduced RFCs for the teams as a tool to discuss designs and proposals to improve our product and work.

With key engineers leaving us, it was important to establish an open culture with teams making independent decisions while focusing on good software design and communication.

As one of the very first RFCs shared, I laid out my admittedly optimistic plan to move away from our legacy monolith app.

The radical 360 plan

I brought my plan directly to senior company leadership triggering a series of discussions in late 2020 to figure out our legacy migration strategy.

In sessions with our CEO Michel, CTO Szilard and head of sales Marlon, it became very clear this would not be just a migration from monolith to serverless. We needed to build the new version of epilot SaaS with a radically different design, starting from a new flexible data layer to power the diverse set of use cases our customers were asking for.

We called it epilot 360. And it would be my job to build it.

From the start of 2021, we established a new strike team to head the effort of building 360. I doubled as the leader of both the platform team and the new 360 team.

The 360 plan was radical. We would completely rebuild our main portal interface using frontend microservices and embed the old product into the new portal as a microfrontend to provide backwards compatibility while we rebuild and migrate features one by one in the new serverless world.

Getting the old monolith to work in the new portal was our first big technical hurdle. We developed a "headless" mode for the old app, where the UI was modified to fit into the new portal layout and implemented secure communication with the parent 360 app to pass information like the current user's session to the legacy app.

From the data side, we continued to split up the legacy SQL database into serverless microservices. We introduced Hasura to act as the gateway to synchronise data between the legacy database and our new serverless services.

By March 2021, we had migrated our first module to 360 using the new flexible data model and had shipped a new portal microfrontend for it.

We introduced Datadog for centralised logging and monitoring, as well as to provide crucial analytics and observability tools for our new serverless product.

Finally in July of 2021, we greeted all our users with a whole new 360 login and portal experience, wrapping the old familiar app in our brand new look with serverless features already replacing many legacy views.

This felt like the first big launch of 360. Although lots of work was still ahead to fully get rid of our legacy app, our users were happy to see us start to deliver on our promises with a new major release of the epilot product.

From here on the rate of new releases in our app would radically increase.

Head of Engineering

In May of 2021, I was officially promoted to head of engineering which meant I was no longer just in charge of the platform team and the 360 team. I was now in charge of the entire engineering team with our CTO shifting his focus away from engineering to product management.

In the meantime we had also ramped up our recruitment efforts. We published our in-house engineering principles and built a new recruitment landing page promise.epilot.cloud to showcase our culture. As a result, we managed to acquire some amazing new engineering talent during 2021. <3

So far it had been mostly just my small team of 4 developers working on 360. Now as the new head of department, I needed to shift the focus of all our by now 25+ engineers and 6 teams to work on the new 360 platform.

This came with some major challenges.

Scaling the team

One of the biggest challenges was the radically different underlying design of 360 with its flexible data model as opposed to our legacy product's traditional static data model.

Teams initially struggled to wrap their heads around the new flexible entities; not least because of my struggle to communicate it properly. Through some much needed visual design support and hands-on experience with the product, we slowly built up an understanding of our new 360 product together.

A major focus of the architectural design of 360 was on modularity by utilising frontend and backend microservices combined with our API First software design approach.

This was to allow teams to work independently giving them the freedom to choose their own tools and ways to solve problems.

We found this approach to be a big success as it kept our projects small with teams motivated to try out new technologies, share and learn from other teams, and quickly change approaches when needed.

After the first few months of teams working on 360 it was clear that we were able to move much faster than ever before with our new tech stack.

We adopted continuous deployment across all teams and quickly surpassed an average of >100 production releases each week. This was a huge achievement considering that back when I joined we were able to deploy only 1-2 times per week during a pre-defined maintenance window!

By the end of 2021, all our teams had shipped major features on the 360 product and we had reached enough feature parity for the first new users to be onboarded to epilot completely on 360 with no trace of the legacy product.

We had an MLP - Minimum Lovable Product!

Things would speed up even more throughout the first half of 2022 with teams getting comfortable with 360, shipping more serverless features to get us closer to the magical feature parity state with our rather large legacy product.

We opened up the 360 platform to external developers and partners with our public developer docs and open source SDK.

In the meantime our sales and customer success teams have been actively engaged with existing customers to help them migrate to 360. More and more users are now adopting the new features each week and we’re learning from them to make 360 even better.

Takeaways from the first year

As I'm writing this post in June 2022, after my first full year as head of engineering and after almost a year of running the main epilot application in production on AWS serverless and frontend microservices, I can't help but ask myself:

Was 360 a success?

While my ego would love to give a resounding "Yes!" as the answer, I can't be fully satisfied until we've fully completed the migration of all features from the legacy app and have successfully shut down the old monolith and moved on with happy users.

We're not quite there yet.

Are we building better software, faster? – Yes! The feedback from our customers is very positive and our product teams are delivering features 10x faster than we were back in 2020 when I joined. We're demoing new features every week now!

Are we on track getting rid of our legacy monolith? – Yes! At the time of writing, we're close to 90% feature parity with more users hopping over to 360 each week.

Was it way more difficult than initially thought? – Heck yes! The people side of things turned out to be much more complex than the technical side – as always.

It isn’t easy to align 5 remote product teams to work on a new product redesigned and rebuilt from the ground up.

It isn't easy to introduce big changes to our tech stack, product, and engineering culture.

It isn't easy to keep our users happy while introducing a new product with major design changes and new concepts to learn.

While none of these things are easy, they are surely worthwhile and we strive to get better every day.

Conclusion

Looking back, I've learned a lot in the past year about leadership, engineering and building products.

With our engineer teams now working well, I've been able to focus my daily work on helping our customers and technical partners hands-on with their adoption of 360. This has really helped me gain better perspective on where the 360 product stands today from our users' perspective.

I've received a ton of feedback; from customers, from colleagues, and from our leadership team.

I’m glad to say things are looking bright for epilot 360!

Still lots to learn and build.

We're hiring @ epilot!

Surprising Performance Lessons from React Microfrontends in Production

Viljami Kuosmanen — Sun, 22 May 2022 12:35:48 +0000

The epilot engineering team stands at 27 developers 1 year after the launch of our rewritten portal built on mostly* React microfrontends.

*Part of our app is written using other frontend frameworks, most notably the sidebar navigation written in Svelte.

Since the initial launch a year ago, our teams have gained a lot of experience running React microfrontends in production using single-spa.

While we expected to face challenges with our new frontend microservices architecture, after solving a few initial problems we haven't hit any major snags with single-spa in the first year.

To my surprise, most issues that have emerged in our codebase are general React pain points not specific to microfrontend architecture at all.

In an effort to share knowledge, I'll address the most common React performance issue we've seen reemerge in our teams in this post.

The state management problem

Here's a really common hook pattern I've seen emerge at one point in most of our React microfrontend projects:

// useFormState.jsx
import React from 'react'

const FormContext = React.createContext()

export const GlobalFormStateProvider = (props) => {
  const [formState, setFormState] = React.useState({})

  return (
    <FormContext.Provider value={{ formState, setFormState }}>
      {props.children}
    </FormContext.Provider>
  )
}

export const useFormState = () => React.useContext(FormContext)

// App.jsx
import { GlobalFormStateProvider } from './useFormState'
import { Form } from './Form' 

export const App = () => (
  <GlobalFormStateProvider>
    <Form />
  </GlobalFormStateProvider>
}

// Form.jsx
import React from 'react'
import { useFormState } from './useFormState'
import { api } from './api'

export const Form = () => (
  const { formState } = useFormState() 

  const handleSubmit = React.useCallback(
    () => api.post('/v1/submit', formState),
    [formState]
  )

  return (
    <form onSubmit={handleSubmit}>
      <FirstFormGroup />
      <SecondFormGroup />
    </form>
  )
)

const FirstFormGroup = () => (
  const { formState, setFormState } = useFormState()

  return (
    <div className="form-group">
      <input
        value={formState.field1}
        onChange={(e) => 
          setFormState({ ...formState, field1: e.target.value })}
      />
      <input
        value={formState.field2}
        onChange={(e) => 
          setFormState({ ...formState, field2: e.target.value })}
      />
    </div>
  )
)

const SecondFormGroup = () => (
  const { formState, setFormState } = useFormState()

   return (
    <div className="form-group">
      <input
        value={formState.field3}
        onChange={(e) => 
          setFormState({ ...formState, field3: e.target.value })}
      />
    </div>
  )
)

Many readers will immediately recognize antipatterns in the above example, but entertain the naïve perspective:

The useFormState() hook is very useful. No prop drilling. No fancy global state management libraries needed. Just native React.useState() shared in a global Context.

What's not to love here?

Perf issues

As nice as useFormState() seems, we'd quickly face performance issues due to components using it having to render on every setFormState() causing unnecessary, potentially expensive re-renders.

This is because we've subscribed all our Form components to re-render on all changes in FormContext by using React.useContext(FormContext) inside useFormState().

You might think React.memo to the rescue, but reading the React docs:

When the nearest above the component updates, this Hook will trigger a re-render with the latest context value passed to that MyContext provider. Even if an ancestor uses React.memo or shouldComponentUpdate, a re-render will still happen starting at the component itself using useContext.

Further, we're unnecessarily depending on the full formState object in all our form components.

Consider:

// formState is a dependency:
setFormState({ ...formState, field1: e.target.value })}
// formState not a dependency:
setFormState((formState) => ({ ...formState, field1: e.target.value }))

At this time, I would consider Context Providers using React.useState to store complex global app state a general React performance antipattern.

However, if React adds useContextSelector (RFC) I am positive the situation could change. 🤞

Lessons learned

Seeing antipatterns like these emerge in React projects even with fairly experienced frontend developers (think 5+ years of React) has lead me to consider performance as a topic that unfortunately requires pretty significant investment to produce quality output when working with React in general.

As always, there is No Silver Bullet. However, our frontend microservices architecture has enabled us to cheaply experiment with different approaches in different teams who have produced quite a few competing strategies to solve form performance:

Use of global state management libraries e.g. Redux, MobX and XState.
Use of dedicated form libraries e.g. react-hook-form
Use of this implementation of useContextSelector
Avoiding controlled form inputs (Leverage the web platform! 👐)

Additionally, thanks to the flexibility of single-spa we've been able to experiment outside of the React ecosystem with frameworks like Svelte and others which has been extremely promising and rewarding for our engineers.

We're hiring @ epilot!

API First: How Teams Collaborate on Software Design

Viljami Kuosmanen — Thu, 17 Dec 2020 14:23:04 +0000

TL;DR:

APIs are the single most useful design tool for teams collaborating on software.
Teams who focus on API design produce better designed software and collaborate more efficiently by parallelising their work around concrete technical contracts.
Thinking of APIs as the core way to add features into software leads into building up a platform for developers to bootstrap new features on top of a library of existing core APIs.

Software Design & Features

As developers we intuitively think of features as the basic building blocks for the software we design and build.

Features are the capabilities our software provides. They answer the question: What does the software do?

We design software around features. Some features are user-facing while many features are purely technical and serve internal purposes to make the system as a whole work.

A software team will add new features, extend existing features and sometimes deprecate and remove old features to be succeeded by new improved ones.

Tests are used to describe features as code and make sure they follow the intended design.

This happens regardless of whether a team follows agile methods and talks about epics and user stories instead of features. These are just tools to help decide which features the software should have and how to prioritise building them.

The Problem

In a small team of up to 5-8 people working on the same project, it's still relatively easy to keep track of all features and come together to make decisions on software design.

This is no longer the case when you scale to multiple teams working on the same software.

Questions like these become increasingly harder to answer:

Which team should be responsible for this feature?
Can I change this feature without breaking other parts of the software?
How was this feature intended to be used?
Could this feature I'm building also be useful to other teams?

Technical Debt

As a result, software teams easily get into situations where the quality of the software starts to suffer; ultimately leading to unmotivated and poorly performing teams having to constantly deal with symptoms of technical debt:

Inconsistent APIs, bad design
Features overlapping in functionality
Tightly coupled features that are hard to extend without breaking things
Having to look into implementation to understand what each piece does
"That one thing only Sam knows about"

How do we fix the problem of multiple teams and individuals collaborating on the same software?

Software Architecture

This is the point we start taking the idea of software architecture seriously. Maybe even hire some people with Architect in their job title to solve the problem.

Software architecture on the surface is about designing and organising the pieces that make up software to make it easier to understand and keep developing.

For our purposes, I would argue it's also about helping teams collaborate and organise their codebases by providing a common language and a set of abstractions to communicate about the design of the system.

Here are a few popular architectural patterns a software architect may suggest to help design your software:

One thing all these patterns have in common is they attempt to abstract the software into a set of smaller, encapsulated units of software.

Most of the units expose an API. An interface to allow other parts of the software to interact with it.

API as a Design Tool

So it turns out thinking about APIs is pretty important for software architecture.

I would go as far as to say the majority of all software in existence is there to provide APIs for other pieces of software. Think software libraries, OOP interfaces, operating systems: software meant to be used by other software with APIs.

In fact APIs are so important, I'd argue API design is the single most useful design tool for teams collaborating on software design.

Not UML diagrams.

Not entity relationship diagrams.

Not abstract architecture diagrams.

API Design.

Why API Design?

If software architecture is about building elaborate systems of features provided by small encapsulated units of software, and those features are mainly manifested as APIs, how can we possibly avoid talking about API design as the centerpiece of the whole design?

That would be like designing a public transportation system without looking at the connections between lines.

Or designing a factory line without looking at how the materials move between the machines.

I imagine those things would be very difficult to do.

API First Design

Next time your team starts working on a feature, after you've drafted a design for how the feature might look like to the user, design the APIs needed for that feature.

Try designing the APIs with code. Make sure not to start implementing it yet. Use types, interfaces or technical API definition standards.

Here are some popular standard tools for designing HTTP APIs:

Share the API design. Include it as part of a proposal or RFC (request-for-comments). Provide examples for how the API would be used. Invite colleagues to give feedback and make suggestions (PRs) to improve the design.

Once the comments stop coming in, run with it. This API will now become part of the beautiful software you're building together.

Efficiency

In addition to producing higher quality software, teams who focus on API design first are able to work more efficiently by parallelising work around the API contract.

Because APIs work like contracts, they can have a future start date. A promise that this API will exist in the software at a later date.

Even before a team starts implementing an API, other teams can already start building new software depending on the technical definition of the API.

Mocks can be used to test the software even before a single line of code has been written to implement the API.

Changes to the API can be proposed with concrete PRs. This is a lot easier while the API only exists as a design with no implementation.

This is a really powerful way to parallelise work and avoid blocking dependencies between teams.

Example of an API First Workflow

Let's say we have two product teams working on an Ecommerce app, both with their own backlogs and user stories they wish to deliver.

Team Alpha plans to add a price notification feature that lets users know when pricing has changed on products they saved in a watchlist.

Team Omega plans to add a new improved search feature that provides product recommendations for users based on their purchase history and products they have saved in a watchlist.

Both teams need a watchlist feature to implement their end-to-end user stories.

A Story of Collaboration

Team Alpha starts working on their price notification story by producing some UI wireframes alongside API designs as OpenAPI definition files in a git repository.

The designs are included as part of a company internal RFC document describing the solution for price notifications.

Omega team views the RFC and notices that Alpha's proposal includes the watchlist feature they need but the Watchlist API is missing a way to fetch all saved products for a user, a feature needed to come up with the recommendations for Omega's search feature.

Omega then makes an adjustment in the OpenAPI definition adding the missing feature to the API design and submits a PR, which gets merged after review from Alpha.

As both teams are happy with Alpha's proposed design, Alpha gets to work on their price notification feature.

Meanwhile Omega submits their own RFC proposal with designs for the improved search feature using the Watchlist API definition as part of the solution. After some collaborative iteration on the RFC document, Omega is ready to continue with their own feature.

Both teams work on their respective features in parallel, mocking the Watchlist API to test out their features.

Later, Alpha informs Omega that the Watchlist API has been implemented as part of their sprint. Omega tests the newly implemented API and is happy to see their search feature now working as designed.

Using this workflow, both teams efficiently collaborated on the design of the software using RFC documents to describe their solutions, including technical API designs to enable working with version control and Pull Requests, both teams working in parallel to complete their user stories.

Platform of Features

As teams focus on APIs first, they end up building up a platform of features.

Focusing on well designed APIs to build end-to-end features, the APIs will implement platform features that can be leveraged by other teams working on other parts of the software.

Smart companies will start building up a Developer Portal, a central repository for the technical API definitions and start providing tools and well written documentation to consume platform APIs. Maybe even release language-specific SDKs to make consuming versioned APIs as easy as installing software packages.

At this stage you have a level 4 API First engineering organisation (see: 4 Levels of API adoption) with teams able to iterate fast by collaborating on the software design on a detailed level, constantly expanding the platform with new APIs and tools to build and consume them.

This is a good place to be. 💙

Forem: Viljami Kuosmanen

We Made Coding Agents Actually Reliable By Fixing One Thing

The Context Problem

Why Compression Matters

Why This Matters

How to Build Your Own

What This Enables

The Constraints Are Disappearing

Start Simple

Dealing with Pushback to Product Engineering

The Ideal vs. the Reality

The Friction Is Normal

Earn the Trust

Why Bother?

Our Leverage

The Sauna Epiphany: How I Got Product Engineering Wrong

I overengineered it

The Wake-Up Call in the Sauna

Boiling It Down to 3 Essential Questions

1. What's the problem?

2. For who?

3. Why is this important?

Leverage Your Team—They're There for a Reason

Taking Action: Start Asking the Right Questions Today

Join the Conversation

Bullshit Tech Roles (satire)

The Product Engineer Checklist

How to Think Like a Product Engineer

1 Understand

1.1 Who's the user?

1.2 Who's the customer?

1.3 What's the market?

1.4 Ask Why

1.5 What do we already know?

2 Craft

2.1 Am I proud of what I'm building?

2.2 Does the product feel good?

2.3 How do I get there faster?

2.4 Teamwork

3 Growth

3.1 How do I measure the success of my work?

3.2 How do I maximise the impact of my work?

3.3 How do I stay ahead of the curve?

4 Product Vision

4.1 What’s our North Star?

4.2 How does my work impact the overall design of the product?

4.3 What's the ambition level?

Product Engineer Mindset

What is a Product Engineer?

Not Just Coders, but Builders

The Counter to Hyperspecialisation

AI: The New Playground

Combining Product Thinking and Technical Execution

The Manifesto: A call to Product Engineers

After 3 years: Bets I made as Head of Engineering rewriting the epilot SaaS – worth it?

Doubling down on AWS & serverless

Microfrontends with single-spa

Using React without a framework

RFCs

API First

Continuous production deploys from main

Not using feature flags from the beginning

Hiring product-focused senior engineers remotely

In-house DevOps / platform team

Weekly Demos

Open-Source Engineering Principles

Hasura

DynamoDB as our go-to database

AWS OpenSearch service

Datadog for observability & monitoring

Datadog for incident management & post-mortems

Datadog for browser tests

Redshift Serverless

Microservices

Conclusion: Was the rewrite a good idea?

What's next?

Comparing REST, GraphQL & tRPC

GraphQL

REST

tRPC

Continuous production deploys from `main`