Forem: Ryan Peterman

Moving Writing to developing.dev

Ryan Peterman — Sat, 22 Jul 2023 16:56:56 +0000

I've been writing weekly for DEV for 4 months now (22 posts). Distribution is two orders of magnitude lower here compared to my Substack so I'm going to stop posting here and just post there instead.

If you're looking for my writing, you may be interested in subscribing to my Substack.

Thanks for reading,
Ryan Peterman

5 Books Every Software Engineer Should Read

Ryan Peterman — Sat, 15 Jul 2023 04:16:43 +0000

👋 Hi, this is Ryan with this week's newsletter. I write about software engineering, big tech/startups and career growth. Thank you for your continued readership, we welcomed almost 2000 new subs last week 🙏🤯

Note: There are no affiliate links in my book list. Enjoy!

The goal for my book list is simple. I wanted it to cover the 20% of reading that you need to do to get 80% of the skills you need on the job.

To make sure this list is high quality, I cross-referenced my recommendations against three sources: Hacker News mentions, Reddit mentions, and Amazon reviews. From my research and experience, here are the top five books every software engineer should read.

Introduction to Algorithms --- "CLRS" (link)

This book is the most famous algorithms & data structures textbook in the world for good reason. It covers everything you need to know on these topics.

When I read it in college for my algorithms class, I was impressed with how applicable it was for interview prep. All Leetcode questions are applications of the knowledge in this book.

It goes further than just Leetcode though. A basic understanding of common data structures is important on the job. Besides, its popularity and ratings speak for themselves:

#1 most mentioned book on /r/programming
#2 most mentioned book on /r/cscareerquestions--- second to an interview prep book (CTCI)
One of the top all-time mentioned books on Hacker News
Average 4.6 star rating on Amazon (2341 ratings) --- much higher than I would have expected for a textbook

Clean Code (link)

Engineers must maintain and modify software to handle business needs. This is much easier to do when code is well organized. This book covers how to write maintainable code.

The knowledge in this book is what helps me write high-quality code and provide useful feedback on code reviews. Given how valuable this book is, it makes sense why it's so popular and highly rated:

Top 10 mentioned book on /r/programming, /r/cscareerquestions, and /r/ExperiencedDevs
One of the top all-time mentioned books on Hacker News
Average 4.7 star rating on Amazon (5722 ratings) --- even higher average ratings than the last book

Designing Data-Intensive Applications (link)

System design is a critical skill for experienced developers. This is because senior developers are responsible for their team's end-to-end systems. Reading this book will help you build modern, scalable systems. I highly recommend this book if you're looking to expand your system design understanding.

Average 4.8 star rating on Amazon (3893 ratings) --- highest among all the books I recommend
Most mentioned book on /r/ExperiencedDevs --- makes sense that this subreddit would be the one talking about this book most
One of the top all-time mentioned books on Hacker News

On Writing Well (link)

Writing well is valuable for software engineers. Top Hacker News posts (ex1, ex2) and figures like Paul Graham agree. Yet, engineers don't get any training on how to write and have to figure it out on the job.

On Writing Well should help; It's a famous book focused on how to write nonfiction better. I recommend this book over The Elements of Style (the other famous book on writing) because this one gives you general guidelines rather than specific rules to memorize. It's helped make my writing clearer. Not to mention that it has great ratings and is often mentioned on Hacker News:

Average 4.8 start rating on Amazon (4872 ratings) --- tied for first among my recommendations
One of the top all-time mentioned books on Hacker News

How to Win Friends and Influence People (link)

Last but not least, if you only have time to read one book, it should probably be this one. The title is a little strange but hear me out.

Just about everything at your job involves other people. You won't go far if you're difficult to work with even if you're talented. The better you are at working with others, the easier it is to get things done. This book is a great guide for soft skills. The popularity of this book matches my recommendation:

It is the most-mentioned book across all of Reddit
Has the most Amazon reviews by far at 111,401 reviews with an average of 4.7 stars

To show you that this recommendation comes from personal experience, here's my copy of the book. It was a gift from my Dad and is older than I am. It's also a bit worn since I've read it a few times:

Join 6500+ software engineers from companies like Google, Meta, and Amazon who receive new posts and support my work on Substack.

Thanks for reading,
Ryan Peterman

Threads' Invisible Armor

Ryan Peterman — Fri, 07 Jul 2023 18:47:22 +0000

👋 Hi, this is Ryan with another edition of my weekly newsletter. Each week, I write about software engineering, big tech/startups and career growth. Thank you for your continued readership, we crossed 5,000 readers this week 🙏 🎉

The recent viral Threads launch got me thinking about how tech companies prepare for major events. For instance, how do you think Amazon prepares for a 50% traffic spike on Prime Day? There's a lot of work that goes on behind the scenes to make sure these services can handle these traffic spikes. If everything goes well, users shouldn't notice anything. Here's what this hidden operational excellence looks like.

Conservative Projections

First, we need to figure out how much traffic we're expecting. Accurate predictions are tough; the goal here is to get a rough estimate. In general, we should overestimate how much traffic we expect to be better prepared.

There are many ways to get these projections. One of the most common ways is to use past data along with your product's current growth rate. In most cases, we'd have folks on the data science side come up with a sensible estimate.

Shadow Traffic Testing

Now that we have an estimate, we can test how our system handles the load we'd expect. My favorite way to test this is to duplicate existing production traffic in a sandboxed environment. These "shadow" requests will exercise the system but will not write to any production databases. That way, we can monitor system health metrics like throughput, latency, and utilization under increased load. There are two reasons load testing like this works well:

Easy Setup --- We piggyback off existing scale and code paths. This makes it simple to set up, despite the complexity involved in confirming these "shadow requests" execute in a sandboxed environment.
Simulates Production Inputs --- Since we copy existing production traffic, we exercise a wide variety of inputs in proportions that match real traffic.

This "shadow traffic" testing is also called "replay traffic" testing by Netflix. See the section where they mention "load testing"; it's a great read.

When your system is at peak load, that's the best opportunity to test any graceful degradation mechanisms. If your system has any knobs that can shed optional load, turn them on and monitor how much they help. You don't want to wait until launch day to confirm these mitigation tactics work.

People Preparation

The most underrated part of preparing for an important launch is making sure your team is ready. This means we should update runbooks and arrange oncall shifts in advance. Oncalls get fatigued if your system is under load into the night. Having shifts helps distribute the burden among the team.

Lastly, you'll want to set up some preemptive communication channels in case things go wrong. This is critical if your infrastructure spans many relevant oncalls. If something happens you don't want people to waste time finding the right place to escalate in.

Digital products can have viral growth moments (e.g. Threads). It's important to capture that growth. Infrastructure preparation ensures that users get a reliable experience during business-critical launches. Otherwise, your product could end up like BeReal, which lost users due to lagging and crashes at its peak.

Join 5000+ software engineers from companies like Google, Meta, and Amazon who receive new posts and support my work on Substack

Thanks for reading,
Ryan Peterman

Resilient Systems Through Retrospection

Ryan Peterman — Fri, 30 Jun 2023 19:07:11 +0000

👋 Hi, this is Ryan with another edition of my weekly newsletter. Each week, I write about software engineering, big tech/startups and career growth. Send me your questions (just reply to this email or comment below) and in return, I'll humbly share my thoughts (example past question).

Every breakage is an opportunity to make our systems more resilient. After leading hundreds of incident reviews, I realize that each review can be reduced down to three simple questions. In this post I'll go over why these questions are exhaustive and what common tactics come from answering these questions.

Systems break, repair them with gold (kintsugi)

Incident Timelines

When our systems break, we think through how we can reduce the impact on our users next time. We can reduce these discussions down to three questions because every breakage has the same high-level timeline:

Users have a degraded experience from when the problem starts until it is mitigated. Therefore, to minimize impact there are only three options:

Detect faster
Mitigate faster
Prevention

These options motivate the three questions we ask in every incident retrospective.

How can we detect this problem faster?

We can't fix problems we don't know about. In areas with less sophisticated alerting, breakages can go unnoticed for days until a user report raises awareness.

Almost all discussions on faster detection revolve around how to enable automated alerts that notify us within 30 minutes of the problem starting. This means adding logging we can alert on, if it doesn't already exist, then writing an alert for it.

In rare cases, it's difficult to find a signal that is stable enough to alert on. Something to be careful of since noisy alerts can be worse than having no alert at all.

How can we mitigate this problem faster?

Once we know there is an issue, the priority is to involve the right people and diagnose it. Common tactics to help with faster diagnosis include making debug logs richer and easier to analyze.

Next, your team will work on deploying the fix. To mitigate faster, we need to think through the deployment cycle for the platform that broke and see if there's a way to get the fix in faster. The best case scenario is if there was a feature flag in production that could redirect traffic within minutes. When that isn't the case, we focus retrospective discussion on how add something like that for the future.

How can we prevent this problem from happening again?

This question is my favorite because it is the most impactful. I always aim to leave retrospective discussions such that, if we complete the agreed upon followups, the incident can't happen again.

There are many tactics that work here but I'll just name a few categories:

Catching bugs earlier --- introducing tests, static analysis, canaries, typing
Building self-healing systems --- adding retries or backfill mechanisms
Making bugs impossible --- removing dependencies, refactoring the code, adding fallbacks

These three questions are the fundamentals of incident retrospection. Remember them and they'll serve you on any team.

Even if you don't break anything yourself, I'd still recommend you get involved in your team's retrospective process. It's impactful and a great way to learn system design because each discussion has a debriefing of the system architecture and why it broke. Thinking through how to improve the system is a great exercise to build your technical skills.

I've run hundreds of retrospectives for my engineering org. If you have anything to add I'd love to discuss. It's something I've been passionate about ever since I broke prod for the first time.

Join 4600+ software engineers who receive new posts on Substack and support my work.

Thanks for reading,
Ryan Peterman

Engineering Career Stories: Jordan Cutler

Ryan Peterman — Sat, 24 Jun 2023 01:22:33 +0000

👋 Hi, this is Ryan with another edition of my weekly newsletter. Each week, I write about software engineering, big tech/startups and career growth.

I'm starting a periodic series of guest posts I'm calling "Engineering Career Stories" that feature engineers with noteworthy career trajectories. The goal of each story is to give readers insight into what has worked so you can learn from others' experience.

Today's post comes from Jordan Cutler, who is a front-end specialist that grew from junior (L3) to senior (L5) in 2 years at Gusto. His story has learnings that work well for growth to senior.

For more from Jordan, you can follow him on LinkedIn. He also writes a great newsletter, High Growth Engineer, about software engineering career growth.

It was my first week at my first job and I broke production.

I also just started this job after wondering if I would even get a tech job given that I didn't get a return offer as an intern at Twitter.

This is the story of how I went from that --- to the youngest Senior Engineer at Gusto among 200+ engineers.

Before we start... a quick glossary:

I'll use FAANG leveling to describe the transitions. I started out as a junior engineer (L3) and was promoted to senior engineer (L5) after 2 promotions.

2019 --- L3 SWE on Financial Products

Just starting out, I 100% had imposter syndrome. Not getting the return offer from Twitter really messed with me and made me feel like the same thing would happen again and I'd get fired after a few months.

The main lesson I learned from not getting the return offer is that it's incredibly important to be a contributing member of the team and to show you are a "sponge" for growth from the people around you. At the time, I was so focused on my intern project that I didn't take feedback and wasn't a great team player. I even pushed back on some Staff engineers' suggestions as an intern.

So starting out at Gusto I made some major changes to prevent that from happening again by:

Asking for feedback a lot from my manager and teammates
Scheduling pairing sessions with everyone on the team and really embracing Gusto's culture.
Making the team better any way I could, like documenting what I learned and improving onboarding for future members, recommending process improvements to my manager--bringing solutions and doing the work for him

These changes laid the groundwork for me to perform well at my level. On top of that, one thing that really helped set me up for promotion early was taking initiative on problems that affected the entire team.

For example, I noticed a very slow part of our process that was frustrating for all the engineers on the team. We'd need to spend at least 10 minutes loading our local dev environment with data to test our changes every time. I remember going to my manager with a Google doc of an idea for speeding up the process to 5 seconds instead of 10 minutes. It was the messiest 1-page Google doc you could imagine but it got the point across.

My manager took that to our PM, got it prioritized, and because I had suggested it, they let me lead the project. It was an amazing opportunity to lead a project on my own, work cross-functionally with other teams, and provide a big productivity boost to our entire team. After completing it, I presented ‌it in our company all-hands which helped me gain visibility across the company. The CTO even came up to me afterward saying it was a great presentation, which was mind-blowing 🤯 for me at the time. The tool proved to have massive value and was used for years.

The lesson here is that I probably wouldn't have had this opportunity so early on if I wasn't looking for ways to make the team better. Or if I had only stuck to my assigned work.

Having that experience helped my manager make a promotion case for me that got me to L4 within 1 year.

2020 --- L4 SWE on Financial Products

After the promotion the first year, my manager made it clear early on that I shouldn't expect a promotion so soon again.

This kept my expectations in check but I still was determined to do as best I could.

A primary driver for my growth this year was the way I got feedback from my manager. Every 6 weeks we'd check in on the expectations of the next level. We color-coded the expectations with green, yellow, and red based on my strengths and areas of growth. By doing this, my manager was able to easily find opportunities to help me turn the reds to yellows and yellows to greens. If all the attributes of the next level were green for 3--6 months, then we'd look to make a promotion case.

Following this process, I looked for ways that I could help the team while also lining up with the expectations of the next level. I'd definitely recommend something similar to this process for anyone looking to get clear feedback for growth and having their manager looking for opportunities for you.

Looking back on the promotion to senior, I think there were 3 major contributors:

Continuing to excel, ship, and lead team project work successfully with minimal bugs. Overall, I became very reliable.
Becoming a domain expert so that teammates could come for advice and expertise, even those that were leveled above me. I was a GraphQL and frontend subject matter expert familiar with multiple product domain areas.
Making the team more effective through proposed initiatives or improved team processes. I led an API re-architecture that was adopted across multiple teams to colocate database ORM queries. Additionally, I tried to make on-call more enjoyable by updating processes around error tracking and fixing root cause issues with persistent annoying error notifications.

Becoming a domain expert is one of the top pieces of advice I would give to anyone looking to get to the senior level. It helped me tremendously.

The way it happened was that our mobile team was looking to adopt and try out GraphQL. I saw it as an opportunity to get involved in something new and interesting, so I asked my manager if I could reimburse a GraphQL learning book and a Udemy course, and I began my learning. I read the book overnight and learned a ton about API design using GraphQL.

Reading that book allowed me to be more involved in technical discussions and provide meaningful suggestions. Because of that, my team trusted me to redesign all our APIs using GraphQL. This gave me a new ownership area of the code and made me a subject matter expert in a new technology area we were exploring.

Once you're a subject matter expert, tons of opportunities will come your way. You'll be asked for help with adopting new patterns, teaching others through presentations, and get asked to lead exciting projects.

All of the above happened for me. And it made making a case for the second promotion to Senior (L5) much easier. As a senior engineer, you're expected to have an area of ownership.

The final lesson to share is to be the main driver behind your career growth. The people around you will help, but you should be the one seeking mentorship, getting the answers you need from your manager, and learning from others what your areas of development are. Even though I was scared and unsure if I would get it at the time, the promotion to Senior only came because I sought what was needed and pushed for putting myself up for promotion. Be the main driver behind your growth.

Conclusion

That promotion to Senior came about 2 years ago now. I went through a team swap to a more frontend-focused role and left Gusto about a year ago to join Qualified as a frontend specialist. My next goal is to get to Staff Engineer, which I'm hoping I can reach within the next few years.

As you can see, I wasn't able to keep up the back-to-back promotions but it goes to show that everyone's career path is different. So I encourage you to take my story, find what resonates and apply what makes sense for you in your situation.

Thank you, Jordan, for sharing this story with us. For more from Jordan, follow him on LinkedIn, and check out his newsletter, High Growth Engineer, a weekly newsletter with actionable advice for software engineers. Have a great weekend 👋

If you find my Substack valuable, share it with a friend, and consider subscribing if you haven't already

Thanks for reading,
Ryan Peterman

Finding Staff-Level Scope

Ryan Peterman — Sat, 24 Jun 2023 01:14:46 +0000

👋 Hi, this is Ryan with another edition of my weekly newsletter. Each week, I write about topics in software engineering, big tech/startups and career growth. Send me your questions (just reply to this email or comment below) and in return, I'll humbly share my thoughts (example past question).

My biggest gap going from Senior to Staff Software Engineer was in finding staff-level scope. My manager paired me with a few mentors to fix this. I took tons of notes and followed their advice, which helped me get promoted to the Staff level in 2 halves. Here's what I learned.

What is Staff-Level Scope?

Engineering work meets the Staff bar based on a few high-level characteristics:

Complexity - Solving problems that senior engineers can't
Impact - Wins span more than just your team and achieve a significant part of collaborative goals
Workstream size - Many engineers involved and roadmap spanning quarters

To become a Staff Engineer you need to find work that fits these criteria and then deliver it. Otherwise you can't grow to that level no matter how hard you work. We'll go over two approaches for how to find staff-level scope.

Finding "Top Down" Opportunity

It is part of the engineering manager's job to know what the top problems their teams are facing. Because of this, your leadership chain is likely aware of some staff-level problems. To find potential work you can ask your manager or skip manager about their top-of-mind problems and opportunities.

This approach is easiest since there is already alignment on the work's importance and potential business impact. Sourcing work in this way also has the added benefit of building trust which can lead to future opportunities.

For example, my director was aware of my work since I'd been consistent and reliable on past projects. Because of this, he gave me the opportunity to lead two org-wide initiatives that contributed to my promotion.

Finding "Bottom Up" Opportunity

What if there is impactful work that your management chain isn't aware of yet? You have the opportunity to convince people that the work is important and then see it through.

This approach is riskier since there's an extra step in convincing others. However, it's the more common path I see for engineers to have unusually fast growth. It also helps you build the skill of influencing others.

Although riskier, this "bottom up" scope has one major benefit in that it's permissionless. No one needs to trust you to hand you the scope; you create the scope yourself.

My promotion to the Staff level came from a "bottom up" workstream. I championed investment in compute efficiency at the right time for our org. This led to massive compute savings wins I wrote about on Meta's engineering blog. We were lucky enough to have Zuck himself write about the savings too.

Finding staff-level scope is hard. Often, such scope may not exist on your immediate team. When this happens, think outside of your immediate team (e.g partnerships or within your larger engineering org). That is a key behavior difference between the Staff and Senior levels.

If you have any questions, feel free to reply to this email or drop a comment with your thoughts. I will reply to every comment as usual.

Join 4000+ software engineers who read my Substack and support my work

Thanks for reading,
Ryan Peterman

Job Hunting In The Layoff Era

Ryan Peterman — Fri, 09 Jun 2023 22:36:56 +0000

👋 Hi, this is Ryan with another edition of my weekly newsletter. Each week, I write about topics in software engineering, big tech/startups and career growth. Send me your questions (just reply to this email or comment below) and in return, I'll humbly share my thoughts (example past question).

Layoffs have happened at all major tech companies except for Apple at this point. Because of this, I've received messages from a few of you considering leaving your company. In this article, I'll go over market data I purchased from Crunchbase Pro and LinkedIn Premium along with my thoughts on finding a new company in the layoff era.

Current Job Market

It's no surprise that it's harder to find a job across the board. There are fewer roles available and more competition for them. FAANG-like company hiring is limited to senior IC roles and a small amount for backfilling attrition. Despite news of doom and gloom among larger companies, there are still some smaller companies growing at a breakneck pace.

Here's a public Google Sheet I typed out by hand containing headcount growth and fundraising rounds for some example high-growth startups.

As you can see, there are still many companies growing like rocketships (e.g. OpenAI +45%, Notion +60%, Figma +101%) within the last 6 months. In addition, some companies have raised even in the current down environment (e.g. Anthropic, Adept AI, OpenAI), which means they have a strong enough narrative and metrics to support growth still.

These high-growth companies have total compensation comparable to what you'd expect at FAANG as well. I pulled the median compensation for a few examples from levels.fyi to illustrate:

Picking the Right Startup

Startup and public company job searches differ significantly. Public companies have equity that is priced by the market every day. If a recruiter says your offer contains $100k of RSUs, you'll receive that amount barring market fluctuations. Also, you don't need to question if the company will still be around in the next few years. However, startup job searches require much more homework. There are a few key signals you should look at if you're considering joining a high-growth startup:

Fundraising History - If a company fundraised in the past six months, it shows growth potential even in the down market. This fresh funding also increases how long the startup can survive which helps with job security. Some are even saying that these startups are more stable than FAANG-like companies in the current environment. Recent fundraising also means that the company's valuation is more likely accurate. Avoid joining a company that raised an overvalued round in 2021 but is worth much less in reality now.

Headcount Growth - Another important signal to pay attention to is recent headcount growth. Since these are private companies, it's difficult to get an exact measurement but LinkedIn premium insights get you pretty close. If you don't want to pay for LinkedIn premium, you can take a look at the spreadsheet I shared earlier. Not only does this measure opportunity for career growth, but this is also one of the top signs a startup is doing well and will raise at a higher valuation later. Take it with a grain of salt though since anything can happen in startups.

Company Stage - Startup compensation and risk vary a lot depending on the stage the company is at. I wrote about compensation differences between startups and big tech here, but the summary is that smaller companies have less liquid, more volatile equity. You'll want to make sure that the company's stage matches your risk appetite and that the cash portion of the compensation is enough for you in the worst case.

I hope this research was helpful for folks working at FAANG-like companies interested in learning more about high-growth startups. If I missed anything or if you have any unique job-hunting insights, please share below for others to benefit. Happy to discuss with you all in the comments as usual.

Join 3800+ software engineers who receive new posts on my Substack and support my work.

Thanks for reading,
Ryan Peterman

Maximizing Impact in the Layoff Era

Ryan Peterman — Fri, 02 Jun 2023 16:53:35 +0000

Most big tech companies have done layoffs aside from Apple. We need to change how we work now that we have fewer teammates. Here's how to have more impact in the layoff era.

Run Towards The Fire

Fewer engineers mean more fires and unsolved problems. It's natural to want to avoid these, but these problems are opportunities for growth.

Identifying problems is a good start. Taking the initiative to report bugs or escalate incidents is helpful. Sometimes teams lack visibility into the severity of a problem, so you can have impact by helping others understand how serious it is. But what is more helpful is being the one to fix it. If you can be the person who consistently does this, you'll develop a reputation that will grow your career.

The skill of solving ambiguous problems separates junior from senior engineers. Junior engineers solve well-scoped tasks, while senior engineers drive resolution for unclear problem areas without any task-level guidance. Each of these unsolved problems is an opportunity for growth to senior levels.

Be Scrappy

Peers that you depended on will no longer be at the company. To make progress on your work, you could ping their teams and ask when they'll staff that work with someone new. This can add weeks to your project timeline. Or, if the ask is simple, you could dive in and do it yourself.

This is what I mean by being "scrappy". Own your project outcomes and do whatever it takes to make progress. If that means that you perform your own data analysis or work in unfamiliar codebases, so be it. This is more efficient than delegation for simple tasks. This is because delegation comes with the cost of communicating the work and waiting for others to pick it up.

Over time this scrappiness will make you a faster engineer. You can avoid the round trips in waiting for others to do things for you if you can do simple work outside your domain (e.g. unfamiliar codebases, data science, data engineering).

Prioritize Ruthlessly

Now that our teams have less bandwidth, it's more important to prioritize. We must only take on work with the highest impact and be clear on what we can't staff. For road mapping, we need to determine the return on engineering time before committing. This sounds obvious, but when there are plenty of engineers it's natural to take on "nice-to-haves" without thinking twice.

If a project isn't impactful enough, we need to drop it and communicate that decision. Leaving work in the purgatory of "eventually" can waste a lot of company time if collaborators are invested in the outcome. For instance, my team once waited months for something to be done by a partner team. It's fine that the project wasn't done since it wasn't a high priority. The real issue is the time wasted in communication for both sides. Every update from their side was a non-update. We spent time checking in and advocating for the work when in reality it wasn't a priority for the other team.

Although the cloud of layoffs hangs over us, there is a silver lining. Problems for the company can be opportunities for the individual. Those who solve these problems will have outsized impact and grow a lot.

I've received messages from a few of you considering leaving your company since growth has stagnated. Hopefully, this post answers part of your question. If you're still interested in looking around, I'll be writing about the current job market next week.

Join 3600+ software engineers who receive new posts on my Substack and support my work.

Thanks for reading,
Ryan Peterman

Generalist or Specialist?

Ryan Peterman — Fri, 26 May 2023 14:21:21 +0000

Q: As a software engineer, is it better to be a specialist or a generalist?

This is one of the questions I get asked the most. I remember I had the same one when I first started working. I wanted to make sure I was going in the right direction but had no idea what each path looked like. Now, I understand the tradeoffs between the two options. Here's what you should keep in mind.

View From The Top

Both specialists and generalists exist at the highest levels of software engineering. They have similar levels of impact; they just work in different ways.

Generalists tend to get work done through others and lead large cross-org workstreams. These behaviors require them to have strong communication and leadership skills. Also, since they work through delegation, they scale themselves by growing others. Their impact comes from leveraging groups of engineers.

Specialists have a similar level of impact by using their domain knowledge. They solve problems that few others can in areas where technology gives them leverage (e.g. databases, distributed systems, AI). Specialists influence others indirectly and contribute to the technical strategy of their domain.

Since both paths support growth to Staff+ levels, what are the distinguishing factors to consider for career planning?

Practical Differences

Each role requires different skill sets. Generalists should have strong system design and communication skills to be able to lead large initiatives. Meanwhile, specialists tend to rely less on others and must hone their domain expertise.

There are also clear differences in role mobility and promotion prospects. By definition, generalists can fit in more roles than specialists can, which can help keep options open. Not to mention that the generalist skill set aligns well with what is required for engineering management. On the other hand, promotions to the Senior Staff (IC7+) levels are more common for specialists since the complexity of their domain can prove they can solve problems no one else can.

The differences in role mobility and promotion prospects shouldn't matter much until Staff+ levels. A senior engineer that has spent a lot of time focusing on databases can still have similar impact in a different domain. Likewise, generalists with the right opportunities should have no trouble getting to the Staff level.

How to Pick

Since there isn't a clear winner, I'd pick based on the work you enjoy and what your strengths are. These two factors reinforce each other. If you lean into your strengths, your success will lead to more enjoyment. Similarly, the more you enjoy your work, the more time and energy you'll spend which will develop your skills.

Consider your career planning as an explore-exploit tradeoff problem. At first, you should explore and try a little bit of everything. This sampling period will help you learn your strengths and what type of work you enjoy. After some time, you can then exploit this knowledge to pick a specialization or to stay a generalist.

For example, when I first started, I was most motivated by impact. I wore a ton of hats and did whatever my team needed most. Through this process, I sampled a lot of different work and realized I enjoyed being a generalist. At the same time, I received feedback that communication was a strength of mine. This experience made me confident that being a generalist is right for me.

It's important to consider, but I wouldn't stress much about this. Specialist and generalist behaviors aren't mutually exclusive. I've worked with strong engineers that exhibit both. Once you identify as one, you aren't pigeonholed into it. Although I'd say I'm more of a generalist, I've done some specialist work as well by working closely with domain experts.

Since this question comes up a lot, I'd like for this post to be a comprehensive reference material for others. If you have any follow-up questions, feel free to drop them in the comments. I will reply to everyone as usual.

Join 3400+ software engineers who receive new posts on Substack and support my work.

Thanks for reading,
Ryan Peterman

Optimizing for Opportunity

Ryan Peterman — Fri, 19 May 2023 15:56:29 +0000

Career growth past the staff level can often seem out of reach. We know that to succeed we need both ability and opportunity. But when it comes to advancing past a certain point, access to the right opportunities becomes the limiting factor. To better understand how to optimize for opportunity, I studied the career paths of 11 people who were at least engineering directors or principal engineers (IC8+) at Meta. Here's what I found.

Engineering Directors:

These directors had a similar story: they worked on a product that was a top company priority, grew a ton, and had outstanding business impact. Their careers grew as quickly as their teams did.

They optimized for opportunity by leading teams focused on mission-critical projects. Some intentionally sought out top priorities, while others became involved by chance. Either way, working on top company priorities gave them access to more resources. In one director's case, she had direct time with top executives and dedicated executive coaching when she needed it.

They also invested in building the necessary skills to take advantage of growth opportunities. The skill set they needed changed as their teams scaled. They mastered prioritization and delegation based on need. One director's thought on prioritization stood out:

"It's often more impactful to take something from good to great rather than from poor to passable."

Product Engineers:

These product engineers' growth also came from working on products that were growing and having tremendous business impact. Along the way, they developed organizational trust from their strong track records.

These engineers had more agency and control over opportunity than their managerial counterparts. They could pick projects that interested them and use prototyping to realize their ideas. One engineer's thought describes this well:

"As engineers, we have the unique ability to just build our ideas. Taking the abstract and making it concrete advances discussion; nothing beats experiencing a product yourself. I've always looked for simple ways to prototype sooner."

These engineers optimized for opportunity by establishing trust through past success, which granted them first access to future critical projects. Also, these engineers were often proactive in working with the most skilled people at the company. One of the engineers even made it a point to go from project to project with the most skilled designer at the company.

Infrastructure Engineers:

These infrastructure engineers all solved high-leverage problems which improved large parts of the engineering organization. Their work had a compounding effect since they made improvements to the tooling or underlying infrastructure every engineer uses at Meta.

These infrastructure engineers seemed to have the most agency in creating opportunities since their impact didn't depend on product growth. Across the board, each engineer champions the quote "code wins arguments". Prototyping and proving their ideas allowed them to create opportunities. None of them mean this as a way to solve problems single-handedly, but use it as a tool to get consensus. One of the engineers summarized this:

"If you want to go fast, go alone. If you want to go far, go together".

They optimized for opportunity by putting themselves in a position to get lucky. They were proactive in chasing the top problems for the engineering organization. Oftentimes, these engineers worked on these problems outside of their team's scope in their spare time. They also invested in continuing their engineering education by pairing with domain experts and reading technical books.

Summary and recommendations:

All 11 of these people were lucky to be working at an incredibly successful, growing company. Growing companies are much more likely to have opportunities for career growth since "a rising tide lifts all boats". If your current company doesn't have that growth potential, you may want to consider switching. Aside from joining a growing company, these people did three things that increased their chances of finding the right opportunity.

First, they sought out business-critical projects that excited them. It's not enough that their projects were a top priority, but also it's important that they were passionate about their projects. It motivated them to do higher-quality work and put more time in which led to better outcomes. If you are not passionate about what you are working on or it is not critical to the success of your company, talk to your manager or consider switching teams.

Second, they were intentional about working with the most talented people they knew. This helped them achieve more through their collaborations. Not to mention this increases the chances of future opportunities since talented people are often working on impactful projects. To do this, factor in the talent of your peers when choosing what to work on and who to collaborate with.

Lastly, they all honed their skills and delivered high-quality work. Their skills increased the chances that they succeeded once the right opportunity presented itself. Collaborating closely with domain experts and reading "subject-killing" resources are two ways to develop your skills. Most domains have a "subject-killing" resource (e.g. High Output Management for eng management) which will tell you most of the critical information you need to know. Also, their track record helped them have access to more opportunities down the road. Always hold a high bar for the quality and polish of your work. It will pay off over time.

Extraordinary career growth isn't something that many will achieve even if they have the necessary ability. Growth at the highest levels requires access to increasingly scarce opportunities so it's not something you should expect. However, you can increase your chances with these behavior changes. Hope this is helpful, enjoy the ride and good luck.

Join 2500+ software engineers who receive new posts on my Substack and support my work

Thanks for reading,
Ryan Peterman

Investing Your Career

Ryan Peterman — Sat, 13 May 2023 15:30:37 +0000

Compensation varies depending on the stage of the company you choose to work for. You can think of your compensation package as an investment. Let's compare the expected value, variance, and liquidity of this investment in a big tech company versus a startup.

Expected Value

People often say that big tech companies pay more because they give higher salaries and larger equity packages. But what if the startup offers comparable total compensation? Often, people value startup equity as near worthless, which isn't accurate. While it's true that any one startup's equity is unlikely to be worth much, its expected value is not near zero.

AngelList plotted venture fund performance across the industry here, which shows that the year with the worst median case performance (2013) had an IRR of 17.5%. Although only a rough estimate, this proves that startup equity is expected to generate a positive return over a large enough sample size. I'd even argue that some startup equity has a comparable expected value to big tech stocks. Otherwise, why would people invest billions of dollars in venture capital when that money could have gone into big tech stocks instead?

In addition, top venture funds often have a competitive advantage because there is often a "rich get richer" dynamic in private markets. Information is less accessible, so funds with stronger networks and access to that information will have a consistent advantage. Top startup deals are competitive and often go to the strongest venture funds because an investment from the top venture firm can signal success and attract future employees and investors. Therefore, while big tech offers have higher expected value in most cases, strong offers from startups with funding from top VCs can have comparable expected value.

Variance

Variance is a much clearer differentiator between big tech and startup compensation. The smaller the company the larger the variance in the value of the equity. This can be either an asset or a liability depending on your personal situation. For most people, the high variance from startup equity is a liability since you may never see a substantial payout given you can only work at a limited number of companies in your career. However, the high variance of startup equity can also be an asset for others who want to take the chance to break into a new social class. With big tech compensation, you'll be well off, but you will never be truly rich (e.g. +$30mil net worth). If you want a chance at riches, you need investments that have higher variance.

Liquidity

Liquidity is where big tech compensation really shines. Equity from big tech is as good as cash once you receive it. Startup equity is often not accessible until a startup is at or near an exit.

One interesting byproduct of this is that big tech equity can be a strict superset of startup equity. Imagine every time your big tech stock vests, you sell it and invest that cash into startups. Since the total value of a big tech equity package is often higher than what you'd get from a startup, you'd own more equity in startups this way than if you just worked at them. This can be a decent strategy if you want exposure to the high variance returns of venture capital but don't want to put all your eggs in one basket.

The main limiting factor here is deal flow. The hottest startups aren't going to let a random big tech employee invest in them. AngelList has made it easier to invest in startups, but you need to pay 20% carry typically so that'd cut into the margins of this strategy. Also, it's unclear if you'd even be able to access the absolute top deals through AngelList since it's so competitive to get capital into top startups. Working at a hot startup directly gets you access to deals you wouldn't otherwise have access to as an angel investor. In addition, your deal flow will improve if you work at a startup since your network is more likely to start sharing deals with you to co-invest in, especially if you are a founder that has ties with your investors.

Big tech compensation makes more sense for most people. That doesn't mean that there aren't some hot startups that have better-than-expected compensation packages. If you get a strong offer from a top startup and you don't need the liquidity for some near-term finances (e.g. paying off a loan or buying a home) then it's reasonable to consider joining a startup.

In reality, the decision to join a company is about much more than just compensation. Even if compensation is critical for you, focusing on where you can get the most skill and career growth is a much more effective strategy for maximizing compensation. This is because senior-level compensation packages dwarf entry-level compensation packages no matter what company you join.

Join 2500+ software engineers who receive new posts on Substack and support my work

Thanks for reading,
Ryan Peterman

Tools of the Trade

Ryan Peterman — Fri, 05 May 2023 14:14:17 +0000

"Give me six hours to chop down a tree, and I will spend the first four sharpening the axe." — Abraham Lincoln

It’s easy to get caught up in the urgency of your main project work and neglect tooling and automation. Setting aside some time to optimize common workflows will help you have more impact in the long run.

Leverage & Satisfaction

Developer tooling has significant impact because of leverage. It’s a productivity multiplier for everyone around you. Imagine you build a tool that 100 engineers are using. Even a small 1% improvement in their productivity would save time equal to having an extra engineer around.

Aside from the potential impact, building developer tooling is satisfying. You make your life easier by automating tedious tasks. It’s also rewarding to see people around you move faster because of your tools.

Understanding Tradeoffs

However, we shouldn't automate everything since there are tradeoffs. Investing in tooling will initially slow progress on our main projects. We should only work on tooling if the incremental gain is higher than working on the problem directly. If done right, it should look like the “theory” panel in the XKCD comic below:

The benefit of tooling and automation comes from how much time we can save. Here's a simple way to think about it:

Total Time Saved = Task Frequency x Time Savings

Tasks that are frequent or expensive have the highest potential benefit. XKCD has a great table showing how long you should spend based on the total time you can save across 5 years:

How To Maximize Your Impact

We should always look for ways to add value immediately. The faster we can make a tool available, even if it isn’t perfect, the faster we’ll be able to get value out of it. Releasing a minimum viable version also has the added benefit of letting you iterate on feedback. Avoid working on nice-to-haves like a clean user interface or handling edge cases at first.

Once you have something useful, distribute it to maximize leverage. This might mean that you package the tool and create a small writeup so it is easier for others to use. From my experience, increasing adoption also improves the tools since engineers who use them often contribute. This creates a virtuous cycle since the more people who adopt the tool, the better it gets, which in turn drives more adoption.

It is easy for our main project work to cannibalize work on tooling. We must set aside time for tooling work to prevent this. There are many ways to do this, but you might consider scheduling a team-wide focus week or hackathon focused on tooling. Allocating dedicated time works much better than hand-wavy guidance to spend 10-20% of your time on tooling improvements.

If you have any past tooling success stories, I’d love to hear them! Feel free to share in the comments on my Substack; I'll reply to everyone as usual.

Thanks for reading,
Ryan Peterman