Forem: Pauline P. Narvas

A Field Guide for Sabotaging Developer Productivity: The Art of Code Chaos

Pauline P. Narvas — Thu, 21 Dec 2023 15:00:00 +0000

In the ever-evolving landscape of tech companies, where the battle for developer productivity rages like an unending storm, a new series emerges – a satirical guide that delves into the art of sabotaging developer productivity. This guide is not just a collection of stories; it's a field manual for those who dare to tread the path of controlled chaos within the realms of software development.

Our satirical journey begins with the legendary tale of Carlo Gustafsson, a name that echoes through the halls of developer history. In the 1970s, this pioneer of productivity sabotage executed a masterstroke, one that would reverberate through the decades: he replaced all tabs with spaces in every changeset. This seemingly innocuous act threw developer tools into disarray, spawning a conflict that would divide developers into two factions – the Spaces and the Tabs.

The Spaces, led by Gustafsson, waged a relentless campaign, transforming every tab into a space with the zeal of a crusader. This act wasn't just about preference; it was a statement, a line drawn in the sand (or perhaps, in the code). The Tabs, feeling the heat of this unorthodox warfare, struck back, subverting the space indentation and turning it back into tabs reclaiming 3 bytes for every tab reestablished! Thus, a cold war was born, not of nations, but of keystrokes, one that persists in companies worldwide.

As a reader intrigued by this history, you might be wondering: How can I, too, become a maestro of developer productivity sabotage? Fear not, for this guide will unveil methods that have been shrouded in the annals of tech lore.

Volume 1: Discontinuous Disintegration

Dive into the world of developer productivity sabotage with the first masterful strategies designed to unsettle even the most stalwart of coders. The first prong of this devious plan involves a cunning manipulation of the continuous integration (CI) system. Envision a scenario where developers are perpetually grappling with a test suite that behaves like a capricious trickster, sprinkling random failures and incomprehensible delays into their workflow. This method's genius lies in its stealth; a seemingly benign requirement stipulates that all PRs must pass the CI build before merging. However, the road to achieving this pass is littered with mysterious and unpredictable obstacles. By introducing these erratic challenges, this strategy aims to erode developers' patience, provoke hasty and ill-considered decisions, and ultimately cultivate an environment rife with errors – a surefire recipe for plummeting productivity.

The second prong in this arsenal of disruption takes psychological warfare to new heights. It mandates that every PR must be rebased from the most recent commit on the main branch before merging. This rule might appear straightforward, but as the frequency of commits to the main branch escalates, so too does the complexity and exasperation associated with struggling to get a passing CI build. Some may speculate that organizations could evolve to trust their developers to merge changes, even amidst the chaos of flaky tests. Yet, this underestimates the resolute nature of DevOps, who operate as the hidden puppeteers in this intricate dance of coding chaos. These two prongs, when combined, form a potent strategy that sows doubt and disarray, turning the once orderly world of software development into a labyrinth of frustration and inefficiency.

As this series unfolds, we will delve deeper into this field guide, unveiling tactics and strategies that have shaped the landscape of developer productivity – for better or for worse. Stay tuned, for this is just the beginning of a journey into the heart of developer chaos, a path few dare to walk, but many will find morbidly fascinating. Welcome to the Field Guide for Sabotaging Developer Productivity. Let the games begin.

Common bottlenecks that impact developer productivity

Pauline P. Narvas — Tue, 19 Dec 2023 14:49:52 +0000

So you want to improve developer productivity? This is what really works

Through my work at Gitpod, I’ve engaged with developers from a diverse range of companies and industries. This experience has provided me with a unique perspective into the factors that impact developer’s productivity, both individually or as part of a team. Instead of going through your typical theoretical frameworks to measure productivity like DORA and SPACE instead today, I want to get really practical and go over the most common challenges for developer productivity I encountered and go over actionable ways to overcome them.

We’ll cover everything from improving your own individual developer flow state with time blocking, increasing your developer feedback loops with tools like hot reloading and why it’s important to standardize to reduce cognitive load.

Individual developer's challenges

There are productivity challenges that any developer will encounter - even if they are working solo:

1. Interruptions and flow state

Developers often work best when in a 'flow state', delivering quality work quickly. In this state, they're deeply focused and highly productive. But frequent interruptions can break this flow, hurting both quality and quantity of work.

Hot fixes:

**Edit your status: **turn off Slack notifications and add a status indicating you are in flow state. Pro tip: leverage something like Clockwise which can automatically slot focus time in your calendar and automatically mute your Slack notifications and change your Slack status for you.
Snooze: turn off all notifications on your Apple devices via the do not disturb feature
Leverage social pressure: get a meeting room or jump on a call with one or more trusted colleagues. Set a fixed time for how long you want to concentrate on a specific task, and share the outcomes each of you wants to achieve. Then collectively work on your tasks, and at the end of that time share which outcomes you were able to accomplish. Because you committed to doing something publicly, you are more likely to follow-through and shut down interruptions. And because you are in a shared room or environment, it reduces the tendency to procrastinate
Centralize interruptions: find a way to centralize interruptions, such that only one person takes the brunt of interruptions and others are free to focus. For example, if there are constant interruptions to engineers due to customer or product questions, have the on-call person be the first responder to all of these. This way, the load is still shared as the on-call person changes, but everyone else gets more focus time. Practically, this can be done via an “ask team X” Slack workflow that pings the current on-call person from that team.

2. Iteration speed (slow inner loop)

The speed at which a developer can change, build, and test code is crucial for productivity. Quicker builds allow for more changes in a single day. Slow builds aren't just inconvenient—they can greatly impede progress. Consider a scenario where a change-build-test cycle takes 10 minutes, with half that time dedicated to building. If you work undisturbed for 5 hours, you can do 30 cycles a day. Reducing build times by just two minutes saves an hour each day or adds 7 additional cycles. That's 21 hours saved per engineer every month!

(Not so) Hot fixes:

More power: opt for larger machines or switch to cloud development environments (CDEs) that offer more resources
Hot-reloading: can you use hot-reloading to instantly reflect code changes in a running service or application? This could nearly eliminate build times!

3. Cognitive load

Developers need to keep a mental map of their code's structure, dependencies, bugs, and more. This mental effort can slow progress, especially with complex projects. If a codebase has unclear boundaries, many interacting components, or confusing names, the mental load further increases.

Hot fixes:

Standardize the development environment: switching to CDEs can result in a lower cognitive load to manage a complex development environment - because the environment is codified. This means that one person can set this up, and others can “just” use it. In their day to day, they only need to think about code changes, not the environment where these changes are made in. Of course, this will not fix all problems that come with a very complex and interrelated architecture - but it does help reduce some of the cognitive load.

4. Developer environment maintenance

A developer's environment isn't static. Operating systems update, tools evolve and sometimes clash. Maintaining this takes time—time not spent writing code.

(Not so) Hot fixes:

Leverage CDEs: Switching to CDEs means that the developer environment is now standardized and is set up to work every time. That means that once set up, developers can focus on writing code, not updating dependencies. Switching between branches or projects is simplified to starting a new workspace, eliminating the need for constant updates to local dependencies and toolchains. Furthermore, updates to dependencies and tools can be managed centrally, ensuring that every new development environment is automatically up-to-date, streamlining the workflow and enhancing overall efficiency.

Team-level challenges

When developers collaborate and share work a new set of challenges emerge:

5. Slow feedback loops (slow outer loop)

When more than one developer is involved, collaboration is essential. They co-own the codebase and need feedback. However, slow feedback loops, whether from a sluggish outer loop (i.e. CI/CD) or lengthy PR reviews, can hamper progress and lower spirits.

Hot fixes:

Unblock before doing your own work: schedule fixed time every morning for developers to do PR reviews before they start their own work.

6. Environment inconsistencies

When multiple developers are involved, different development environments come into play. These can vary based on machine setup, OS, developer preferences, and update frequency. This can lead to a code change working on one machine but not another— the "it works on my machine" problem. \

Additionally, there are CI and production environments to consider. These can differ greatly, especially when developers use Windows or Mac, and production runs on Linux containers. This mismatch can reduce productivity, as what works in the development environment might break in production.

Hot fixes:

Standardize your development environments: by codifying your development environments via a gitpod.yml or devcontianer.json file you can make sure that every developer gets the same environment every time and, when running the environment in a container, “dev” matches “production”. No more “works on my machine” problems.

7. Onboarding onto unfamiliar codebases

A lone developer knows all their code - they wrote it. But in a team or a large company, developers often encounter unfamiliar code. Contributing to an unfamiliar codebase is costly - not just because of the learning curve to understand the code, but also to have the right tools and dependencies set up. An outdated readme.md file usually guides this setup. This initial hurdle of setting up not only hampers productivity, but also discourages developers from contributing to other code bases hurting cross-team collaboration and hampering any inner-source initiative before it takes off.

Hot fixes:

Ask each first-time-user to improve the readme.md of the repo: The repo’s readme.md file usually explains how to set up your development environment. Keeping it up to date can be a burden and is often neglected due to other priorities. Make it a thing for developers onboarding onto a repo for the first time to fix any inconsistencies in the readme.md that they find - e.g. with a CTA at the top. Let’s be honest: This is unlikely to do much, but it is better than nothing :)
Codify the readme.md by using CDEs: CDEs are effectively a codified version of your readme.md environment setup guide. They describe the expected development environment as code once, and then each instance (or workspace) of that environment will behave exactly the same. Someone unfamiliar with a code base will have the same exact experience as someone that has worked on it for years - all by just pressing a single button. Any changes made to the development environment are automatically picked up by any new instance of it.

8. Costly context switches

Reviewing a PR or “quickly” switching from feature work to fixing a bug means switching context - not just mentally, but also in the development environment. You need to checkout a different code branch, build it, and install dependencies. This can take time, especially with large codebases and varied tool requirements. It's time spent waiting, not coding.

Hot fixes:

Find ways to reduce the number of “switches”: if you do PR reviews in the morning only, you might be able to reduce the number of times you switch contexts because you do not need to switch back and forth as often, and can review PRs that require similar environments all at once.
(again) Leverage CDEs: with CDEs, you can easily do a PR review without having to make any changes to the environment you are working in. Instead, just spin up an entirely new environment just for that other branch or PR you want to check out, do what you need to do, and then switch back.

Conclusion

The role of the developer, whether working individually or as part of a team, is marked by challenges that extend beyond simply writing effective code. The challenges that arise compound when developers need to collaborate with others. Cloud Development Environments (CDEs), where a developers environment is standardized and readily available in the cloud, are a holistic solution to all of the challenges listed above. CDEs are worth a deeper look if you can identify with the challenges listed - find out more by downloading our white paper.

Confessions from sales: my playbook for creating a bulletproof business case to invest in a developer tool

Pauline P. Narvas — Thu, 14 Dec 2023 13:46:47 +0000

Let’s face it: most software engineers struggle with business cases. Most engineers focus too much on solutions, jargon, and details, and not enough on hard-hitting and straight-talking business value. The best engineers, however, know how to speak the language of the business and translate technical projects into real business impact.

So, why is it that some engineers thrive where others struggle?

Now, I know what you are already thinking: “That’s bold! How do you know engineers are so bad at writing business cases?” And the short answer is: because I watch engineers struggle creating business cases every day. And it’s my job to help engineers craft business cases that compel, speak to technical leaders like CTOs and CISOs, and ultimately get projects off the ground. Let me explain…

Gitpod has a free tier, open to the entire world. Which means that we’ve seen over 1 million developers on our platform. Rather unsurprisingly, we get asked every single day: “I love Gitpod! How can I convince my boss that I can also use Gitpod at work?”. Whilst I absolutely love hearing this question, I can’t help but feel how this question has flaws. The question is not really “how can I convince my boss?”. The question is: “How do I show how Gitpod can help my company to achieve its goals?”. And this applies to any developer tool, also.

Today, I’m going to give you the playbook I use when helping Gitpod customers create business cases. I’ll use Gitpod as the example here, but you can easily apply this playbook to build a rock solid business case for any product or tool.

Step 1 - start with the “why?”

When it comes to embarking on any new initiative, it’s imperative that you start to think very deeply about your “why”. What needs are you hoping to fulfill with this project? What is the compelling future you are creating for your organization? Or, what urgent need are you solving?

If you have not read “start with the why” by Simon Sinek - it’s a book on leadership and persuasion. Sinek argues: “People don’t buy what you do, they buy why you do it” and the same is exactly true when it comes to building business cases. If you are having issues you know a cloud development environment can solve for - your peers and managers are simply going to want to know “why should we purchase this?”

You should have this thought fully fleshed out. Does this help achieve overarching business goals such as shipping product faster? Does it help with your developer experience initiative? Does it help with a mandate around security? Does it help strengthen the story of your cloud first-approach to infra and tooling? All of these questions help tie back to why purchasing this tool impacts goals that your leadership are thinking about.

Another way to help tie back to business goals is by explaining the problems and pain you are feeling today. Use this as an anchor to justify how this tool’s solution is going to help with the given problems. We will circle back to this as it’s part of the business case - but before doing the work - you need to understand how to articulate the why.

Step 2 - research and requirements.

Google is your friend. Forums are your friend. G2 is your friend. Reddit has good info. Find out who the top vendors are and research them deeply to determine who you may want to partner with. How should you decide?

There are 2 ways:

Social proof (reviews, ads, word of mouth)
Trying the free version of the product to get an idea of what that product is like.

After you understand what vendors stand out - make a short list. Looking at 12 vendors is a waste of time - in most cases only 3-5 vendors will be best in class, the rest will be column fodder. There is no reason to subject your team to look at 12 vendors just to find out the top vendors were the most popular for a reason.

Then comes the hard part - creating technical requirements.

For CDE vendors, you should understand if they:

Support connectivity to cloud and on-prem resources
Work with all your IDEs and editors
Are extensible with your existing tooling
Support your SCM of choice
Connect to your security tools (SSO, VPN, etc)

For CDEs, this is not just a tool for developers, platform, security, devops or engineering leadership. It is actually a powerful platform that will affect all of them - and you should absolutely find out what is important to each stakeholder group and invite them to look at your requirements list. I call this out as this is the case for many developer tools now, and leveraging the wants and needs across cross-functional stakeholders, will only strengthen your case.

Step 3 - engage vendors.

You have a reason for why, you understand your company's technical requirements across stakeholders and you know who the top vendors are in the space. It is time to (take a deep breath) - talk to salespeople.

As someone who has been a sales person for over a decade - I promise we are not all bad. Remember - the more open you are with your account executive - the more they can help you and that is what they are there for. They will run a process called discovery to better understand the current state of your business, your pain points, internal structure and tech stack to determine if you are a good fit for their offering.

If you come prepared with your technical and business requirements - this will tremendously help them assist you - doing this work beforehand will also allow you to stay objective in your evaluation and not get a vendor to suggest a checklist that only aligns to their offerings.

Vendors should be helpful to you in this process - but they can only be helpful if you give them relevant information and access to the team to make a consensus driven decision. Taking a first call with just yourself is fine - but be prepared to invite other folks from your team to the demo.

Step 4 - shortlist, test and start the internal procurement process.

You are now at the proof of concept stage. Something that we refer to as a proof of value here at Gitpod - as successful evaluations consider both technical and business requirements.\
Before you dive into a POV with anyone - it would behoove you to understand your internal procurement process - especially if you have not done this before. A typical procurement process will need the following:

Alignment across all teams that the issue is real and works within their boundaries.
An executive buyer (someone who can sign a contract and has access to budget for a tool).
A security review - your team likely has a questionnaire for software vendors to complete.
Contacts from procurement and legal that will need to be involved to negotiate on terms and commercials of the agreement.

You can never start this process too early. Vendors can assist you in navigating this process, but only introduce vendors you like and trust.

In the POV stage, it is important to test our use cases that solve for your “whys”. Deep technical testing, combined with commercial evaluation will set you up for success. You will need to carve out employee time for these projects. As a rule of thumb - you should never, ever, POV with more than 3 vendors at once. It will be very clear what products are technically sound and which products are not a good fit within the first few days of getting it up and running.

During the testing phase we recommend devoting at least 8-10 dev hours to each vendor you are evaluating. It’s important to give each platform a rigorous trial to make sure that you’re comfortable with installation, setup, and usage.

For CDEs specifically - there should be a big emphasis on developer experience - as you are changing a workflow devs have known their entire career. Get feedback from them - as end user adoption is the most critical part of a successful implementation. Here at Gitpod - we send out engineering surveys before and after testing to measure results (link content here).

Step 5 build your business case

You should have nailed your “whys” and identified some main use cases that you validated are solved for with your vendors. Once you have identified your emerging winner - you have to build out a business case with that vendor.

In the business case you should have:

A history of the why.
Metrics around how pains of local development are hurting staff today.
A return on investment calculation
A summary of why you chose that specific vendor as preferred and some high level information about them - including price.

Your vendor should help you put the above together. Here at Gitpod, we help our partners with the business case and have a very detail oriented ROI calculator (contact us if you’re interested in a copy) that tells you how much money you will save by implementing a CDE with your data inputs.

Step 6 - enjoy.

Take a deep breath and get ready to implement and solve your problems. Purchasing a CDE doesn’t have to be a lot of work if you partner with the right company and the benefits will make you and the team look great with less work!

If you have any questions feel free to reach out to the sales team at Gitpod.

The cost vs. UX trade offs of designing and operating CDEs

Pauline P. Narvas — Tue, 12 Dec 2023 11:13:20 +0000

Let’s build a cloud development environment

Cloud Development Environments (CDEs) offer significant improvements to developer experience and productivity, as well as security by centralizing source code access. Given how close they are to a company's IP and core operations, for some companies and industries it is sensible to consider running them within the company network and build them in-house. But how do you get started, and what are the likely challenges that you will face? This blog post takes you on a mental journey of turning a simple solution into something that works in practice and strikes the right balance between cost and user experience.

V0: The starting point: always-on VM-based CDEs

It makes sense to start simple - after all, KISS is a well known engineering principle for a reason. Initially, keeping it simple allows you to assess the benefits of CDEs for your company without committing to months-long implementation efforts. The KISS version of CDEs involves providing each developer with a long-running VM in the cloud (e.g. EC2), running open-VScode-server on the VM to develop remotely, and using .devcontainer json file to describe and standardize the development environment. Running in a container on a VM also has the benefit of your development environment more closely resembling your production environment. This solution will allow you to benefit from some of the core benefits of CDEs, namely equitable and consistent development environments. And notably, this architecture is how companies like Stripe, Slack and Uber got started on their journeys to building CDE solutions.

Side note: successfully rolling out developer tools is not trivial. This blog post outlines proven strategies for successful developer tool adoption.

V0.1: Automatic shut down

The fact that the VMs are always on is good for user experience as it allows for quickly jumping in and out of existing development environments with almost no wait times. However, this architecture can become very costly, very quickly. For example, a comparatively small 8 core, 16GB RAM VM with a fast disk (useful because a lot of dev workloads are very I/O heavy) can cost around $300 per month at on-demand pricing. Multiplying this by 100 engineers, this costs $360,000 per year.

Once your CDE solution sees some adoption, it’s sensible to look for ways to reduce cost. The first obvious way is to shut down the VMs when not in use. To do this without losing an engineer’s work you need a reliable way to persist data from a development environment, enabling engineers to pick up where they left off. Allowing engineers to pick up where they left off not only means persisting any working directories where source code may be stored, but also (ideally) also any shell history, environment variables, (ideally) runtime state and more. Without these the developer needs to reconfigure every time they jump into a development environment.

There are several different storage options, ranging from using the instance storage of the VM to using networked storage (such as EBS or S3 on AWS) that can be detach and reattach to VMs, decoupling the VM from the development environment state. Decoupling storage from VMs increases flexibility, because you can attach the state of one VM to another - such as attaching the state of a “paused” development environment to a pooled or “warm” VM and thus increasing startup times (see below).

Another complex aspect is the shut down or timeout logic. When do you shut down a development environment? How do you know when an engineer is ‘done’ with their environment and not working late? There are several approaches here, ranging from hour of the day based shut downs, to user-defined timeouts at each startup, or even automatic inactivity-triggered timeouts. The choice depends on the context, however the latter is the most scalable and user-friendly.

Implementing automatic shutdown of development environments can lead to considerable savings. Assuming a development environment only runs for 10 hours each week day, the abovementioned cost of $360,000 per year would be reduced to ~$104,800 (50 hours per week _ 52 _ $0.4032 p/h * 100 engineers).

V0.2: Improving workspace startup times

This significant cost reduction must be balanced against its impact on user experience. After implementing VM shutdowns, engineers have to wait for an entire VM to spin up whenever they want to write a line of code - at the start of each workday, after timeouts, or when launching additional environments for tasks like PR reviews or testing new features. VM start up times can often take around 2 minutes. And once the VM itself is running, it needs to be setup for development: the devcontainer needs to be spun up and configured, a backup potentially downloaded and unzipped, source code checked out, development tools started, code built and more. If 100 engineers each start their development environment twice a day, and each startup takes 5 minutes, the total daily time lost is 1000 minutes. This is equivalent to over 16 hours, or more than two full 8-hour workdays. Considering an annual wage of $120,000 per engineer, this inefficiency costs around $240,000 a year - about the same as the amount saved by spinning down VMs above! This highlights that improving CDE startup times can be just as if not more effective in saving cost as reducing infrastructure cost. Outside of cost savings via increased productivity, a faster startup time is sure to delight users and keeps them from context switching while waiting.

Development environment startup times can be reduced in many ways. A sensible first step is to always have a pool of VMs ready to go that can be used to spin up a development environment. When an engineer requests an environment, they receive one from the pool. Then a development environment defined in the .devcontainer.json is started and another VM is spun up to keep the number of VMs in the pool constant. This requires you to build logic that manages this pooling. There is a clear trade off between user experience and cost here: increasing the size of the pool increases the chances that there will be a machine ready to go for every engineer, however this also increases the cost. Finding the right balance depends on how much you are willing to pay for a reduction in startup times.

While a pool of pre-warmed machines eliminates VM spin-up time, it doesn’t necessarily speed up the setup of development environments. Further, the tasks that are run when starting a development environment are likely the same for most engineers - meaning the same work is happening many times over. To counter this, it is reasonable to implement logic that prebuilds and preconfigures development environments on top of the pooled VMs. With this, startup times can be reduced to less than 1 minute (depending on the code base) and engineers can immediately start developing when they get their environment. Prebuilding also comes with cost versus UX considerations: you need to decide the frequency of prebuilding to incorporate main code changes efficiently. A balanced approach, like prebuilding every 10th commit, could be an effective solution.

V0.3: Reducing cost even further:

Incorporating the user experience benefits above will unfortunately increase your costs again. For example, a pool of 10 8-core VMs running 10 hours per weekday costs ~$880 month or about $10,400 a year. This is on top of the ~$104,800 that it costs just to run the development environments during working hours. As such, it is sensible to again find more ways to reduce cost.

Engineers don’t always need 100% of the hardware resources of their development environment. Usually 100% of available resources are only needed when building code or running other compute intensive tasks like data science workloads. If several development environments are run on the same VM, chances are high that not every developer is using all of the hardware resources of their development environment at the same time. This means that we can try to add more development environments onto a VM than would theoretically fit (a.k.a. oversubscribing or overbooking). Doing this can further significantly reduce cost, as we can run more development environments with the same amount of resources. This can be done by running them in containers, and running several containers on each VM. Either via Docker or leveraging container orchestration solutions like Kubernetes (and in this case, achieving this by setting requests and limits appropriately).

To further optimize cost, it is important to shut down VMs when they are not in use. However, when running several environments on one machine, you need to wait until the last one shuts down before you can shut down the VM. This again can cause unnecessarily high costs. The solution would be to add the capability to shut down a dev environment on one machine and then spin it back up on another - effectively allowing the running development environments to be defragmented, and the utilization of VMs to be increased.

Depending on how quickly and smoothly this happens, it can impact the user experience and interrupt an engineer in their flow. Again, here is a tradeoff here between UX and cost.

Taking stock

Let’s take stock of how far we got: we created a system that allows developers to fairly quickly spin up an on-demand development environment at a manageable cost. There were many questions that had to be answered, revolving around storage content, timeout logic and duration, architecture choices (VMs versus containers), oversubscribing, prebuild frequency, and the size of the pool of instances that are kept ready. This just scratches the surface of things that need to be considered, but it gives a glimpse at how complex it is to build and operate a CDE solution in house. Some important considerations were not covered:

**Observability: **CDEs can quickly become business critical, and hence a good level of observability and alerting is essential.
**Updates: **Updates are also critical - when there is a patch for a vulnerability in an image being used, how do you make sure that all development environments are updated when they are running all the time? And how do you update them to avoid impacting users? Also, how do you update the infrastructure (e.g. new AMI) without impacting users?

Getting to a sensible spot regarding cost versus UX takes a lot of implementation effort. Just trying things out with a naive implementation is unlikely to create a system that can withstand the requirements of continuous use at reasonable cost. To short circuit this, it is practical to consider vendor-managed CDE products such as Gitpod. You can try Gitpod for free today whether it runs in your cloud account or Gitpod’s.

Rethinking the trade-offs

Here's a thought: what if this trade-off can be avoided altogether? Imagine having development environments that are available exactly when you need them and gone when you don’t, just like opening and closing a laptop, with all processes running precisely where you left them. And all of this without the cost of perpetually running instances and also with almost unlimited development environments.

That's exactly what we at Gitpod are working on right now. Stay tuned as we reveal more about our journey to rethink the cost vs. UX tradeoffs of CDEs over the next few months.

Champion Building - How to successfully adopt a developer tool

Pauline P. Narvas — Mon, 11 Dec 2023 13:41:59 +0000

So you've just bought a new platform tool? Maybe it's Hashicorp Vault? Snyk? Backstage? You’re excited about all of the developer experience, security and other benefits you're about to unleash on your company—right? But wait…

Because, if you've ever rolled out a developer tool, you know: it’s hard. No matter how shiny the tool. If it’s your first time rolling out a tool, I will guarantee that you’ll make mistakes: you’ll push out your tool too early, or too late, which would compromise your timeline. If you misjudge your timing and communications, you risk users having a poor experience, leaving you to fight to win them back in future. These adoption mistakes at best will stall your adoption, but at worst they can completely derail your project, causing it to outright fail.

In this post, you'll learn

why running an 'all-hands' demo too early can hurt developer tooling adoption
why champion-building is your highest leverage for adoption success
best practices for building knowledge, support, documentation and more.

Adopting a developer tool is hard

How do I know? Because I've been there. Back in 2021, I was part of a platform team that was one of the first adopters of Backstage. I've also seen what it takes to move 4,000 repositories to GitHub Actions (spoiler: it wasn't easy). And these days, I help our customers adopt Gitpod, a cloud development environment (CDE) for their engineering organizations to improve developer velocity. It's no surprise that I see many similar challenges to those I've faced in the past.

Over the past couple of years at Gitpod, I've seen the difference between companies that adopt Gitpod quickly and those that take longer to realize the developer velocity improvements that come with a Cloud Development Environment. So, the big question is: why are some companies faster at adopting developer tools than others? Let's explore that today.

Beware of the early 'all hands' demo trap

One trend that I've noticed teams gravitate towards in the early days is wanting to run demos or 'all hands' calls, typically involving as many engineers as possible, with the intention of a ‘grand unveiling’.

Behold! A new developer tool—in all its glory!

However, as is nearly always the case with these 'all hands' calls, a group of engineers will immediately try the developer tool, and briefly after exploring it before thoughts of urgent pull requests will flash in their minds, soon leading to those developers having their focus pulled back to their urgent and pressing needs of the business. So, how do we avoid this fate?

Assuming that visibility is the main challenge and that excitement alone is sufficient for successful adoption. But the reality is that excitement wanes, and the reality of the business's urgent needs soon takes precedence. And business priorities always win.

There's nothing wrong with demoing a developer tool. But it's the timing of these demos and the narrative you craft can be critical to what happens after the demo. Instead of rushing into demos, what I've found most successful is acknowledging that developer tools adoption is a socio-technical challenge, and to which the most effective pattern we see is champion building.

But, what is champion building? And how do you do it?

You need to start champion building

Adopting a developer tool within an organization is not just a technical task, but a socio-technical challenge, necessitating a strategy that involves people, priorities, and technology. The key to successful adoption lies beyond the tool itself; it's about intentionally engaging engineers and leveraging their surrounding ecosystem for effective adoption.

Why it's so important to "start small"

One of the most critical factors to getting a tool off the ground, is building simultaneously: organic, real and true support from your own engineers, whilst also building a few compelling real-life use-cases. It's usually important you build this foundation before widely inviting your engineering organization to use the developer tool that you're adopting.

Ideally, you start by working with one or two teams or use-cases maximum. You want these use-cases to be shining examples of the future you're trying to create with the adoption of your new developer tool and ideally use-cases that solve obvious and painful needs of the business. Nothing wins over skeptics, naysayers and detractors better than being able to say: "Don't think it's possible? Well let's take a quick tour of how X team are doing things".

The steps to champion building for a developer tool look something like this:

Identify internal champions
Build one or two compelling use cases
Expand and scale

Step 1: Identify internal champions

Find motivated, energetic, and ideally influential engineers within your organization. These champions could be senior engineers like tech-leads, principal engineers, or even junior engineers eager to prove themselves. Look for engineers who are dreamers, who believe in the future, not ones that find solace in the status quo. Once you've got a few champion candidates, it's time for step 2.

Step 2: Build one or two compelling use cases

Now that you have your champions, work closely with them, integrating your tool into projects they are involved in where you will see the highest business return. Ensure the use cases you work on are within their sphere of influence and the champion can be available to assist others. That typically means adopting the tool within a project that your champion works on day-to-day.

It is vital these use-cases are close to your champion's influence, and that your champion has time to support others. Because you will undoubtedly have issues, and will need your champions to be present to quickly resolve them. Time to respond in the early days is going to be very important.

Avoid the mistake of being too distant with early use cases. Pull them close. Not just metaphorically, but physically. If you're a co-located company, sit physically with the champion(s) and their teams.

I've seen this done well with the concept of a "platform team rota", where an engineer does a mini "secondment" into the platform team whilst you're getting things up and running.

As you build out these use-cases the main outcomes you're looking for are to:

1. Grow "institutional" knowledge in your engineering organization

2. Build documentation that connects the tool and your company context

Let's break each of these down a little further:

Grow "institutional" knowledge in your engineering organization

You want to deliberately build knowledge depth of the tool you're adopting with your champion(s). Deep knowledge only comes from using a tool frequently, ideally day-to-day. A great litmus test is when you see your champion(s) answering other engineers' questions quickly and without referring to any documentation. Institutional knowledge will be vital later in your adoption as it can significantly reduce the support burden which otherwise will fall on the adopting team (read: you).

Build documentation that connects the tool and company context

Every organization is structured differently. As you build that institutional knowledge, you want to create documentation which will tie together the developer tool to the specific context of your engineering organization. Most developer tools are highly flexible, so it's not uncommon that you'll need to show how to connect a combination of features to solve certain use-cases.

In the early days it's common for teams to "copy" other teams configurations and setup. Make sure to signpost the best-in-class examples of how to use the tool. If you don't signpost examples well, engineers are going to search in GitHub and copy/paste whatever they find first, which can be difficult to unpick later, and could lead to a bad first-time experience.

As you discover tips-and-tricks, document them. Treat each new engineer as an "experiment" to test the effectiveness of your documentation. Pay attention to the questions those engineers ask like: "Can I connect Gitpod to Vault?" or "How do I login to AWS with Gitpod?". The frequently asked question (FAQ) format works best for these types of documentation.

Focus on providing real-world examples and optimize for copy/paste.

Step 3: Expand and scale

If you've successfully completed the champion-building steps above, you should now have:

One or two compelling use-cases of the tool being used in practice
Some good initial documentation, most likely in an FAQ type format.
A handful of engineers who have good institutional knowledge of how the tool works.

You might want to consider quickly surveying your initial use-case engineers to gather quotes. These quotes can be incredibly useful to influence leadership on your return on investment (ROI) in adopting the developer tool. Social proof is the most powerful currency you have to convince future engineers to switch to a new developer tool.

At this point, running the 'all hands' and focussing on driving visibility to the developer tool can work. Ideally, when planning any demos of the tool, you don't present the developer tool yourself, but get your champion(s) to tell success stories.

Steer the focus of the stories so that you are leading with the value delivered. Showcase what you achieved with the tool, not just what you did. This is a great opportunity to give your champions credit for helping you to get this far with your adoption, and you want others in your organization to see those champions as "the people to go to" for when they need help, support and inspiration.

From here it's a case of working through additional teams and use-cases. Continue the same process of building champions, pulling champions close and iterating on their setups. Do it right, and you'll find with each new team you have fewer and fewer issues. Do it right, and you'll see engineers supporting other engineers and adoption will thrive.

You're on your way to a successful developer tool adoption

Hopefully this gave you some insights into champion-building, which is what we have seen to be an incredibly successful model for adopting a developer tool within an organization. If it's your first time driving adoption of a developer tool, hopefully you now have some more tools in your toolbox.

Remember: Identify champions. Build compelling use-cases. Use early engineers to iterate on your documentation. Give visibility to those champions as social proof. Achieve developer velocity.

Before we wrap up, I wanted to takes a moment to pay a special thanks to the folks who helped proofread and provide additional insight into this blog post:

Josh Gavant - CNCF Platforms Working Group
Simon Emms - Lead Platform Engineer
Bryan Ross - Executive Advisor
William Ghelfi - Senior Software Engineer & Gitpod Community Hero

So, as always, thank you ! The tech community never ceases to amaze me with it's generosity.

Self-hosted, not self-managed

Pauline P. Narvas — Fri, 08 Dec 2023 13:01:12 +0000

When you want to introduce cloud development environments (CDEs) to your organization, you have to decide between building or buying, and self-managing or self-hosting. These decisions are more than just technical, they impact every aspect of your development and operational efficiency. So whether you are managing a team of developers or overseeing the infrastructure of a large organization, understanding the nuances of these decisions, especially self-hosting or self-managing, is crucial for your future strategies.

In December of 2022, we discontinued our self-managed offering. After three years of trying to make it work we realized that CDE platforms benefit from not being self-managed. While they can be self-hosted, self-management makes them less effective and more costly. In this article I’ll explain why self-managed CDEs do not offer the experience we wanted to provide to our customers - and what an alternative can look like.

These are hard-won lessons. Giving up on self-managed meant forfeiting seven figures of net-new revenue and eventually led to us laying off 28% of Gitpodders beginning of 2023. Those were never easy decisions. The alternative we created with Gitpod Dedicated, which is still self-hosted but not self-managed, is so much better - and we have the customers to prove it.

CDEs are mission critical

Cloud Development Environments are mission critical software. They are the place in which developers write software, review their work and use the tooling required to be ready to code. When your CDE service is down your engineers cannot work - every minute of downtime is very expensive and frustrating. Therefore, reliability is fundamental for any CDE service. This basic need is well understood by organizations adopting CDEs and often drives their desire to self-manage their CDE installations.

Tight vertical integration is crucial to deliver an experience that adheres to the CDE principles (Equitable, On-Demand, Extensible, Consistent), while meeting basic needs. Hence, full control over the technology stack that powers a cloud development environment is a prerequisite. Any flaw in integrating the various components of a CDE can undermine reliability and security. Consider, for example, workspace content: there’s little more excruciating than losing your work after hours of intense concentration. Any CDE must ensure that your changes are not lost because of state handling bugs or node failures - something which requires significant influence over the underlying infrastructure.

CDEs aren’t merely better VDIs or VMs moved to the cloud. They represent a foundational shift in how we develop software, how we collaborate and what expectations we have towards our development environments. We want to build the best experience possible, and not compromise on the lowest common infrastructure denominator.

Until 2023, Gitpod offered a product inaccurately labeled as “Gitpod Self-Hosted”. At the time, we didn’t understand the difference between self-hosted and self-managed. Self-hosted implies that the service runs on your infrastructure, such as an AWS account, and aids in network integration and compliance. In contrast, self-managed means taking full operational responsibility for the service. The distinction has become increasingly relevant with the growing popularity of Software as a Service (SaaS) offerings.

Self-managing CDEs seems intuitive, as it aligns with decades of engineering ‘best practice’. Engineers are accustomed to maintaining their own development environments and adjacent services, like Continuous Integration. However, running a service close to home differs significantly from managing it yourself. Managing a service, i.e. assuming responsibility for it means overseeing every aspect: components, data storage methods, computing resources, and network setup. In cases of downtime, the burden falls on the operator who is then called out of bed. This level of control introduces demands to CDEs that can conflict with the vertical integration required to provide a great user experience.

Installing self-managed CDEs is hard

Any CDE provider integrates a number of different services and functions. Consider storage: whenever a workspace stops, its content needs to be backed up, often in object storage like S3. Chances are your organization already runs its own storage service, maybe MinIO. Wouldn’t it make sense if the CDE service you want to operate reused that existing MinIO installation? This preference extends to other CDE components. Operating yet another database, no thank you - we already got one over here. Docker registry? Our security team says we have to use Artifactory. Kubernetes? That would be GKE on 1.25, thank you very much. As the team who’s responsible for making sure the most foundational need –reliability – is met, it’s just sensible to rely on existing infrastructure. This also adds a complex set of requirements to any CDE of your choice: it has to support all these existing technologies and systems.

In larger organizations, chances are these existing infrastructure systems are managed by teams of their own. There’s the Kubernetes team, the networking team, the S3 team, and the database team. Coordinating with these teams for self-managing a CDE platform can be a complex task. You’ll need to make sure that what those teams offer aligns with the requirements and capabilities of the CDE software you intend to run - and if not, need time from those teams to ensure they do. We have seen this process take weeks, sometimes months. Self-managing a CDE in the real world can be a complex and time consuming endeavor of organizational navigation.

Kubernetes is the hardest part of this technology stack. Few organizations have a team which truly understands Kubernetes in depth - most, sensibly, focus on operational concerns and methodology. For CDEs that use Kubernetes, like Gitpod, deep integration is essential to deliver expected security and performance. Point in case: dynamic resource allocation. Gitpod has supported resource overbooking for many years, but only recently has this capability found its way into Kubernetes. To make this happen we had to break the abstractions Kubernetes provides and interface with the container runtime and Linux cgroups directly. The same is true for security boundaries, network bandwidth control, IOPS and storage quotas. Such deep vertical integration imposes specific requirements on the container runtime (e.g. has to be containerd ≥ 1.6), Linux kernel version or filesystems, which are unusual for typical Kubernetes-focused teams, but not for CDEs.

While the consumption of Kubernetes software has improved, with increased skill and knowledge among teams and UX enhancements within the community, managing complex Kubernetes software remains a significant challenge. This has led to entire companies being built to attempt to solve this problem.

Frankensteining software

The inherent need to adapt CDEs to pre-existing infrastructure and an innate desire for integration within an existing ecosystem often results in a situation we’ve come to call “frankensteining”. For CDE vendors, accommodating the diverse ways of integrating with various infrastructures is a challenging task.
As a CDE customer, the desire for deeper integration with the CDE platform frequently outpaces what the platforms currently support. This is especially true for customers managing and owning the software. Naturally additional means for authentication using internal APIs are added, new custom features are built on assumptions which break with the next update. That clandestine CDE installation becomes a frankenstein implementation that is hard to update, difficult to operate and near impossible to support.

Operating self-managed CDEs is bad for your sleep

Now, you’ve managed to piece together all the required bits of infrastructure. You’ve gone through the installation and are finally up and running. Developers in your organization love their CDEs: no more “works on my machine” issues, onboarding time is virtually non-existent - as are context switching costs - and you got brownie-points from your security team because endpoint security is less of a concern. In short, the CDE platform you now operate is becoming mission critical.

Much like any mission critical software, maintaining operational excellence is crucial. Observability forms the cornerstone of your ability to guarantee the most foundational need: reliability. Much like the other infrastructure building blocks you probably already have an observability stack in place - and if not you’ll need one now. CDEs sport a myriad of failure modes, due to their dynamism (starting and stopping workloads several hundreds of times a day) and variance (developers run a variety of different applications with different load profiles). Once you’ve integrated the multitude of metrics, imported or built your own dashboards, understood and set up the alerts and hooked it all up to your PagerDuty installation, you can finally sleep well again - until PagerDuty ends your slumber of course.

Every month or so a new update will be available. Considering that CDEs are mission critical, updates must cause no downtime - expected or unexpected. Kubernetes simplifies this process but isn’t a complete solution. Successful updates require cooperation between your team and the CDE vendor to ensure tools and methods, like blue/green deployments or canary releases, are in place for smooth transitions. Ensuring continuous operation of all workspaces, preserving ongoing work, and maintaining functionality of integrations like SCM, SSO, Docker registries, package repositions, can be stressful. Better do this on a Sunday morning.

Frankensteined installations make updates a gamble and these custom configurations deviate significantly from the standard setups tested by CDE vendors rendering them unhelpful. The uncertainty of whether your customizations will remain functional post-update adds to the challenge.

The operational risk and effort of updates incentivizes slow and few updates of your CDE installation. Consequently, staying up to date with security fixes, and offering the latest features your developers crave for is hard. As CDE vendor product cycles become longer, feedback becomes less. It becomes harder for them to provide the best possible experience. This creates a lose-lose situation.

Self-Hosted, but not self-managed

There’s a middle way that balances customers’ desire for control and integration with the advantage of not having to manage everything themselves: the distinction between self-hosting and self-managing.

In November 2022, during an offsite in beautiful Sonoma County we made the decision to end our self-managed offering, previously mislabeled as “self-hosted”, in favor of what we now call Gitpod Dedicated. This was not an easy decision with very direct and tangible consequences. However, years of providing a self-managed CDE offering led us to believe that there must be a better way: self-hosted, but vendor-managed.

“Vendor-managed” solutions have become a standard way to consume software, most commonly referred to as Software as a Service (SaaS). SaaS offers several compelling benefits:

It eliminates the need for you to operate the software, as you can purchase a Service Level Agreement (SLA) directly from the vendor, freeing your resources and ensuring someone else is called out of bed when things go sideways.
It requires minimal upfront investment; you simply sign up and start using the product.
It removes the burden of updating the software, as you automatically receive updates and new features.
The quality and features of the software improve rapidly, as vendors can execute quicker feedback loops and product development cycles in a SaaS model.

More and more companies aim to bring these benefits to a self-hosted model. Database vendors like PlanetScale, data service providers like Databricks or Snowflake, and other developer tooling companies have pioneered this approach. For Cloud Development Environments, this model is nearly ideal, offering a practical balance between control and enjoying convenience.

Self-Hosted, vendor managed CDEs: Gitpod Dedicated

Let’s look at the Gitpod Dedicated architecture to understand how such a self-hosted, vendor managed CDE can look like. First, we need to distinguish between infrastructure and application.

Infrastructure management

All infrastructure is installed using a vendor-supplied CloudFormation template in less than 20 minutes. This template orchestrates the creation of things like a VPC, subnets, load balancer, EC2 auto-scaling groups, an RDS database, and crucially, all IAM roles and permissions.
The customer applies and updates this CloudFormation template, ensuring they retain complete control over their infrastructure.

Application management

The Gitpod application, operating on this infrastructure, is installed and updated through a set of Lambdas deployed as part of the overall infrastructure.
These Lambdas connect to a control plane responsible for orchestrating updates and handling operational aspects like telemetry, alerting and log introspection.

A pivotal aspect of this model is the meticulous separation of permissions, allowing customers to maintain authority over their infrastructure while offloading the operational burden of the software.

Early Adoption

Gitpod Dedicated was developed in close collaboration with early adopter customers, ensuring its architecture would meet the stringent security and architecture requirements of diverse companies. Gitpod Dedicated has since been reviewed - and passed - the most stringent security reviews:

Forward-looking industry leaders like Dynatrace and Quizlet have since adopted Gitpod Dedicated, praising its ease of installation and low operational overhead.
Large financial institutions adopted CDEs by virtue of the control afforded by the Dedicated architecture.

Several customers have migrated from our previous self-managed offering to Gitpod Dedicated. All reported reduced operational effort and some saw up to 93% faster workspace startup times.

We have time and time again seen how simple vendor-managed CDEs are to install. Throughout the many Gitpod Dedicated installations, it’s become clear that reducing the variance of how a product is installed makes the product more reliable and easier to use.

We’re always looking to learn more about how to make the experience of using CDEs optimal while meeting any and all security and architecture requirements that organizations have.

You can trial Gitpod Dedicated today and drop us feedback on what your experience was like.

Internal developer portals aren't a silver bullet for platform engineering

Pauline P. Narvas — Mon, 04 Dec 2023 12:54:38 +0000

A hard-to-ignore theme at KubeCon this year was how many organizations are grappling with the challenges of improving developer velocity and wrangling the increasing complexity of their tooling. To improve the situation, many organizations are now investing in, or exploring the idea of platform engineering as the solution to their developer velocity challenges.

In the absence of a prescriptive guide on how to build a platform, many organizations are turning to internal developer portals as their first investment. Whilst some organizations do achieve success, many are also struggling. The “average Backstage adoption rate is stuck at 10%” reported The New Stack. For inexperienced platform teams, internal developer portals risk becoming a hammer looking for a nail in a scramble to solve for developer velocity.

In this article, we explore the trend towards internal developer portals, and the reasons why many platform teams seem to be struggling with adoption. We then take a look at why our platform tools must use interfaces most relevant to our developers, if we are to avoid needing to resort to top-down mandates to spur platform tooling adoption. Drawing on my past experiences adopting Backstage, and stories from Gitpod, hopefully you’ll be able to avoid similar mistakes and instead achieve successful adoption for your platform.

Caption: "Build an ecosystem, not a wilderness" - slogan from the Backstage homepage.

Backstage first launched in 2020 by the engineering team at Spotify. The announcements caught the eye of the developer experience team I was working on at DAZN. What Spotify were articulating in their messaging felt so painfully similar to what we felt at the time, as we wrangled with trying to support in increasing the developer velocity of a scaling engineering organization, built on top of microservice principles and architecture.

Software sprawl was rife. Duplicated work was happening across many different libraries and packages across the organization. Developers complained of the amount of work and platform knowledge required to launch a new service. Reliability was a top concern, and during incidents, it was often hard to figure out which service belonged to which team, leading to longer than tolerable outages. Leadership could only look on, dismayed.

“Build an ecosystem not a wilderness” was the messaging of Backstage. Which resonated deeply with us, and soon we became early adopters. Since Backstage was announced the market for internal developer portals has exploded, and we see many other tools in the market like getport, cortex, opslevel offering various takes on the same underlying principles of having a centralized, extensible catalog “single-pane-of-glass” into your internal developer platform.

Caption: Five Backstage lessons in 5 minutes - Chris Westerhold, Thoughtworks

However, with the growth of Spotify Backstage, comes a growing emphasis on internal developer portals as a significant piece of the platform engineering puzzle. With many newly minted platform engineering functions risk making the cardinal sin of the platform, which is thinking “if we build it they will come”. As Chris Westerhold of Thoughtworks pointed out at his BackstageCon talk: “you can actually make your developer experience worse” (source)

It’s becoming a trope that every Backstage talk seems to start and end more-or-less identically with an opener like: “we had challenges with developer experience, we started our journey to platform engineering, then we discovered Backstage…”. before coming to a conclusion with some variation of: “internal developer portals aren’t a silver bullet”, or...

“If you build it they will come doesn’t work”.

This trend points to a repeating pattern.

Which makes me wonder: did these platform teams not watch the other adopter talks? Or, do these platform teams believe they are unique enough to not need to heed the warnings from others who have walked a similar path before them? And finally, what is it about internal developer portals that creates such a big challenge for adoption?

Caption: Spotify Engineering Culture.

Spotify’s Backstage feels to have a parallel to the Spotify model, a structure for engineering organizations that Spotify popularized. The spotify model was subsequently adopted by many organizations as the “silver bullet” to their software delivery challenges. However, as many organizations soon found out: what works for Spotify might not work for you.

Just as the human body can “reject” a foreign transplant, engineering organizations can reject “implanted” culture and tools, including developer portals.

I believe a big part of the challenge here is that platform teams make the mistake of jumping too early to the solution. Skipping over the—often difficult—step of building a deep, empathetic understanding for the problem they are solving. However, the realization of the overconfidence always comes too late in the process, typically when the developer portal is unveiled and the adoption numbers don’t grow as you might have hoped. If the agile revolution taught us anything, it’s that we must test our assumptions, and work in small iterations to de-risk.

Another challenge is that when the path of least resistance for new platform capabilities is the developer portal, portals become the go-to option for interfacing into the platform. Portals might be easy to implement, but that doesn’t mean they’re easy for developers to adopt. When adoption challenges arise, the common response is usually that the portal “needs more functionality”. However, what experience has taught me is that adding more functionality into a developer tool seldom solves adoption challenges—as the challenges are often more complex.

What I’ve learned from my time working on platform teams in the past, and now from working on product at Gitpod is how critical it is to

"Leave interface choice to the developer—and never break their workflow”

As articulated by Humanitec in: how to make an enterprise grade internal developer platform.

Because, if you’ve ever tried, you know just how difficult it is to change the habits of your users. And this is particularly true if your users are developers.

Caption: Mechanical Keyboard by Michelle Ding (Unsplash)

Developers are highly technical users, with incredibly specific behaviors that have been honed through their many years of their professional work. Developers are often tuned towards quite specific interface mental models when completing certain tasks.

When it comes to tooling adoption, what I’ve come to learn is:

The sane choice—often the only choice—is to: "meet developers where they are аt"

We need a deep understanding of our developers, and we need to understand their behavioral patterns. User researchers have understood the importance of observing users in their natural environment for some time to inform product decision making, researchers call this practice contextual inquiry. Which is the practice of observing users in their context, to more deeply understand how they perform certain tasks. Why? Because the best indicator of future behavior is past behavior. With platform tooling, we’re often trying to nudge developers towards best practices. To do this, we need to understand developers' current behaviors.

To underscore how critical it is to “meet developers where they are at”, let me give you a few examples from my time working at Gitpod. At the time of writing, Gitpod has over >1 million developers on the platform. In building out Gitpod, we’ve learned a lot about developer experience, and we’ve also certainly made our fair share of mistakes. But what we’ve also seen is that every time we move closer to meeting developers where they are at, we also see a spike in adoption, which is not merely a coincidence.

For context, Gitpod is a “Cloud Development Environment''. Which in simple terms means that when developers are using Gitpod, they don’t work on their own machines, but they work in the cloud. There is certainly a lot more to cloud development environments, in general, but that’s not the purpose of today's blog. All you need to know is that the primary interface to Gitpod is mostly an editor or IDE. And, as we’ve learned over the years, the editor is at the center of the developer workflow—a very sensitive part of the developer experience.

Caption: Header image from From Theia to OpenVSCode Server - A history of Cloud IDEs

The first time we saw one of those significant spikes in adoption was when Gitpod moved from the browser-based editor developed by Gitpod called Theia to VS Code. At the time, this move felt bold and risky. Theia was inspired by and looked incredibly similar to VS Code. But, what we learned in hindsight was how the developer experience simply wasn’t good enough for mass adoption. Developers wanted the full, undiluted VS Code experience they had come to rely on—and not some “diet coke” version. As I mentioned, the editor is one of the most sensitive aspects of the developer experience, and even subtle differences cause distraction and pull developers out of their flow state. Moving to VS Code marked one of the first major upticks in users on the platform, no doubt attributed to the familiarity and convenience of VS Code.

We then saw another similar spike in adoption when we announced our JetBrains integration. At the time we were astounded by just how many users were using VS Code with Gitpod, but were also secretly dying to use their more familiar JetBrains IDE. Again, possibly unsurprisingly, by giving developers literally their interface of preference, we again saw adoption spike. What I learned throughout both of these stories was just how specific developer requirements for interfaces can be. Admittedly, it can be difficult to comprehend this phenomena unless you’ve experienced it first hand, which is why I also believe that many platform teams are often overly optimistic about their adoption numbers when it comes to new developer tools.

I could share more examples, but by now I hope the point is fairly clear. Developers are power users, with unique mental models, behaviors and expectations for their tooling. If hoping developers change their workflows is tied to your platform's success, adoption is going to be a difficult, or even impossible task. Because, the more you diverge from meeting developers where they’re at, the more you’ll need to apply top-down pressure, mandates and send threatening emails to get your developers to use your platform tooling. Which goes against the mantra of most platform teams, who actively want their tools to be so good that developers actively want to use them.

Instead, if we’re looking for true adoption success, we should choose the right interface for the job, taking each tool on a case-by-case basis. Platform teams need to remember how many options we have: building command line interfaces, libraries, editor plugins, browser extensions, slack bots, native apps, pull request apps, continuous integration plugins and more to bring the interface of the platform closer to our developers to support with driving adoption. We need to avoid the temptation to put all of our platform tools into our portal, simply because we have one. That all being said, internal developer portals absolutely do have a very necessary role in our platform engineering stack, but we do need to be careful that portals don’t become a hammer looking for a nail.

If the original goal of platform engineering was to keep our developers in flow, then blindly baking every platform tool into a user interface doesn’t make sense. Instead, we should strive to meet developers where they are at and keep developers in flow.

A special thanks to the folks outside Gitpod who also helped by reviewing, proofreading or otherwise providing valuable inputs that lead to this blog: Roy Ronalds (Software Architect), Fernando Villalba (Platform Engineer, OpsLevel), Himanshu Mishra (Product, Harness), Boris Tane (Founder, Baselime).

4 Simple Ways to Deploy your small website

Pauline P. Narvas — Sat, 15 May 2021 09:45:03 +0000

Over the past few years, I’ve been working on several different projects mostly for fun or my own learning. There was always nothing like spending hours on the development of these projects (which have mostly been websites) then the excitement of deploying to share with the world.

Deploying can be quite overwhelming, mostly because of the vast amount of ways that you can do it. I recall when I was still starting out, trying to publish my projects was another often stressful thing to figure out. Googling can lead you down several rabbit holes, especially if this is your first time deploying anything. Trust me, I know that based on my own experience. 😂

Luckily today it is easier than ever to get your projects online. Different services nowadays not only deploy your website for you but also allows you easily update your changes without thinking much about it. In some cases, a quick push of new code to your main branch is all it takes once it’s all set-up which is awesome!

Here are some of my favourite ways to deploy web-based projects. (Bear in mind, these are mostly small websites.)

Vercel

I recently used Vercel for the deployment of a recent re-build of my portfolio. You can read about my experience of re-writing my portfolio from scratch in Next.is here.

Out of the other ways I’ve deployed my websites in the past, Vercel was by far the greatest developer experience. It was quick and easy to set-up and even easier to deploy any changes after the initial push. I have since moved all my other small projects to this way of deployment just because of how easy it is!

Get started here

GitHub Pages

A popular way of deploying small websites is using GitHub Pages. This was my favourite way of deployment for a couple of years! I first learned it off a Code First: Girls Web Development course back in 2016.

Your website would be hosted directly from your GitHub repository which means that any changes that you need to make moving forward, all you’d need to do is to push your changes to your main branch and your changes should be live in minutes.

Get started here

AWS CodeDeploy

If you like Amazon Web Services (AWS), their various Code services might be the solution for you. CodeDeploy specifically works great with CodeCommit (a code repo, similar to GitHub) but can also be integrated with GitHub, AWS CodePipeline or Jenkins.

I recently used CodeDeploy for a small React project at work and was surprised by how easy it was to set-up. Similar to the rest of the deployment services I’ve listed already, AWS CodeDeploy makes continuous deployment as simple as possible.

A cool thing about using AWS CodeDeploy is the fact that since it is integrated with the vast variety of AWS services, you can do some pretty awesome things. For this small project at work, I set-up SNS alerts that were sent to a Slack channel. This was useful for collaboration because developers on the project could see when a PR was raised and what needed approving.

I also just like playing around with AWS services and seeing how they all work together. I find it super cool 😂

Get started here

Heroku

Similar to Vercel, Heroku has a pretty streamline flow when it comes to continuous delivery. One of my Code First: Girls Python project and a hackathon project was deployed on Heroku! It definitely is another way to maximise developer productivity and support that modern development practice of continuous deployment.

Get started here

And there you have it – just some of the ways that I have deployed several of my small website projects. All of them differ slightly but at the centre of it all focus ensuring that the deployment process is simple for everyone.

Over to you – what are some of your favourite ways to deploy small projects?

Building my Website-as-a-Service project

Pauline P. Narvas — Thu, 13 May 2021 11:54:15 +0000

Building my Website-as-a-Service (WaaS) project

The first time I found out about Terraform was in 2019 during the third rotation of my graduate scheme. At the time, I hadn’t realised how powerful infrastructure as code was but now that I’ve had a good amount of exposure to these technologies, I can’t think of anything better! Spinning up infrastructure in minutes and it all just working (after multiple config changes because nothing ever works first time 😂) is a dream!

One of the projects that I had worked on whilst on that rotation was what we called - “website as a service.” The idea was to give teams a chance to spin up their own static websites, backed by AWS’ Simple Storage Service (S3) using a pipeline. All they had to do was enter some parameters that corresponds to the details of their website for example, domain name.

In my new company, I recently participated in a company-wide Hackathon. The team I had joined had a similar idea that they had in mind, and it may be cheating but I decided to join their team to get my hands in Terraform again after not doing so for a while. With that said, although the solution was similar, it wasn’t exactly the same (as I had to plug things together that worked with our internal tech stacks) and I actually ended up learning some new things — especially on the Jenkins/writing groovy scripts front.

So for this blog post, I wanted to share my solution that I hacked together after reading various documentations and seeing what other Engineers had done (thank you, Google!) I won’t be going over Jenkins part in detail, but will have some pointers for you to get started.

I’ll be re-using code that I’ve seen other Engineers implement - a list of original posts can be found at the end of this blog post.

Setting up

There are a few things that you need to make sure that you have on hand:

An AWS account with AWS CLI access
Terraform installed on your laptop (Download Terraform - Terraform by HashiCorp)

Set up your project

The best part of starting a project - getting set-up and organised with your fresh motivation to do the thing. Relatable? Yeah, I feel ya.

mkdir website-terraform
cd website-terraform

Let’s also make sure that our Terraform version is up-to-date.
terraform version

Initialising Terraform

Coolio. We’re ready! Let’s get writing some Terraform. Let’s create some files!

terraform.tf
dist
    - index.html
    - style.css
    - about
        - about-me.html
AWS-modules
    - s3.tf
    - s3-iam.tf
    - route53.tf
    - variables.tf

In this case, you may want to just copy and paste the Terraform code which is completely fine if you want to get this set-up quickly.

But if you’re a newbie learning Terraform for the first time, I found that typing some Terraform for the first time line-by-line even though I had the code in front of me helped me understand the syntax better. I guess this is true for learning to code in general. Newbie learning hack! 😊

In your code editor (my personal favourite is VS Code), open up your terraform.tf file. This is where we first create our first Terraform configuration! Build Infrastructure | Terraform - HashiCorp Learn Terraform is built up of various blocks, in this case, we have a providers block that states the plugin we are going to use. In this case, it’s AWS because that is the cloud provider that will host all our resources. If you’re familiar with other cloud providers, you can use those too. The required providers is required for latest version of Terraform.

In the provider block, you’ll see a profile and region key/value pair. This refers to which region you will be deploying your resources and which AWS credentials profile to use (usually you can leave this as default, but if you have multiple AWS accounts make sure that this points to the right one.

Remember that it is IMPORTANT not to hard-code any credentials here

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 2.70.0"
    }
  }
}

provider "aws" {
    profile = "default"
  region = var.aws_region
}

module "website" {
  source = "./.deploy/terraform/static-site"
  domain_name = var.domain_name
}

The module block links to all the resources that make up our static website.

In variables.tf, this is where all our variables will live that we reference across our Terraform files! They don’t need any defaults but if you already know what value you want every single time e.g. a region you definitely want to use for all your resources then it’s a good idea to set a default here. Otherwise, you can leave it blank. In our case, it’s best to leave it blank because this will later be parameterised in our Jenkins job.

variable "aws_region" {
  type = string
    default = "eu-west-1"
}

variable "domain_name" {
  type = string
}

Whilst we are here, let’s create a simple HTML website that also has a link to an “About Me” page.

<!DOCTYPE html>
<head>Welcome to my static website!</head>
<a href="/pages/about.html" >About me</a>

<p>This site was deployed via cool Terraform magic</p>
</html>

<html>
<h1>About Me</h1>
<p>I'm a page!</p>
</html>

Setting up your S3 Bucket

Resource blocks are used to define components of your infrastructure. In the Documentation it states that “a resource might be a physical or virtual component such as an EC2 instance, or it can be a logical resource such as a Heroku application. Resource blocks have two strings before the block: the resource type and the resource name. “

In our case, we’re creating an S3 bucket, IAM policy for the S3 bucket and Route53 resource. As you look through the code for creating the different parts of infrastructure as code, try and see if you can understand what the Terraform code is relating to.

Those who have some AWS knowledge may be able to see how a line relates to something you may have seen on the AWS Management Console. This is how I like to think of it when I’m writing my Terraform code!

s3.tf

Creates a bucket that corresponds to the domain name you input (linking back to the variables.tf file)
Adds a policy to the bucket and has an ACL that is set to public-read (so that anyone with the link can access it)
With the website {} block, this S3 bucket now knows that, “Oh, I’m going to be used as a website! Yay!” And maps the index_document in the root folder to index.html as well as the error_document (i.e. what happens when a user goes to your-website.com/random-made-up-page)

resource "aws_s3_bucket" "my-website" {
  bucket = "${var.domain_name}"
  acl = "public-read"
  policy = data.aws_iam_policy_document.my-website_policy.json
  website {
    index_document = "index.html"
    error_document = "error.html"
  }
}

s3-iam.tf

This creates the policy for your S3 bucket. You can read up on AWS policies here.

data "aws_iam_policy_document" "my-website_policy" {
  statement {
    actions = [
      "s3:GetObject"
    ]
    principals {
      identifiers = ["*"]
      type = "AWS"
    }
    resources = [
      "arn:aws:s3:::${var.domain_name}*"
    ]
  }
}

Setting up Route53

route53.tf

What could this Terraform code be creating, I wonder? 🤔 That’s right! A Route53 resource! Again, if you’re familiar with navigating the AWS Management Console, some of these fields may look familiar. In this case, we’re creating an Alias record that corresponds to the S3 generated URL from the bucket we created.
Bear in mind - you do need to purchase a domain for this to work.
Notice how I’ve used the domain name variable here again.

resource "aws_route53_record" "www" {
  name = "${var.domain_name}"
  type = "A"
  alias {
    name =  aws_s3_bucket.my-website_bucket.website_domain
    zone_id = aws_s3_bucket.my-website_bucket.hosted_zone_id
    evaluate_target_health = false
  }
}

Running a Terraform plan / apply

Now for the fun part! Let’s start initiating the Terraform and see all the code that you just wrote in action.

Some of key Terraform commands:

terraform init
This initiates the Terraform based on the config in terraform.tf

terraform plan
This command creates a plan of the infrastructure that you’re about to deploy. It’s a good idea to always run a plan to make sure that you see exactly what resources is going to be created on AWS.

terraform apply
If you’re all happy with what you see in the plan then go ahead and run an apply. This creates all the resources on AWS.

On your AWS Console, you should see all your resources built! Magic, right?

As easy as it was to create your infrastructure, you can also delete it all at once too with…

terraform destroy

🥳🎉🍾 Powerful, right? That is Infrastructure as Code for you! 🎉🍾🥳

Using Jenkins - some pointers

Now I have to put my hand up and be honest here… groovy scripts isn’t my strong point. I also don’t enjoy using Jenkins if we’re being even more real over here. 😆 But as part of this project, Jenkins was in scope - after all, it wasn’t just for me to run it all locally but for it to benefit both technical and non-technical colleagues from other squads too.

I won’t dive into groovy in this blog post (there’s already quite a bit of Terraform here to get your head around) but here are some pointers to think about:

Parameterise the domain name so that it takes in input from users
A simple bash script that uses the AWS S3 Cli to run an AWS S3 sync
Or if your index.html or application build is not committed to a repo (e.g. GitHub or BitBucket) then potentially a way to upload and unzip your application build onto S3 via a File Upload parameter.

Other things to think about

The solution above doesn’t cover using HTTPS, but it’s wise to do so. You could add Cloudfront to front your S3 website! Give it a go. 😀

Woohoo! What a wild ride, right? I hope that this inspires you to dabble in some Terraform. I think that this project was a fantastic way for me to get my head round Infrastructure as Code and other Cloud native technologies to make playing around with AWS much more streamlined and fun. ☁️

Let me know if this was useful to you, happy playing around in da Clouds!

blog

Building my Website-as-a-Service project

Pauline P. Narvas — Thu, 13 May 2021 11:51:24 +0000

I’ll be re-using code that I’ve seen other Engineers implement - a list of original posts can be found at the end of this blog post.

Setting up

There are a few things that you need to make sure that you have on hand:

An AWS account with AWS CLI access
Terraform installed on your laptop (Download Terraform - Terraform by HashiCorp)

Set up your project

The best part of starting a project - getting set-up and organised with your fresh motivation to do the thing. Relatable? Yeah, I feel ya.

mkdir website-terraform
cd website-terraform

Let’s also make sure that our Terraform version is up-to-date.
terraform version

Initialising Terraform

Coolio. We’re ready! Let’s get writing some Terraform. Let’s create some files!

terraform.tf
dist
    - index.html
    - style.css
    - about
        - about-me.html
AWS-modules
    - s3.tf
    - s3-iam.tf
    - route53.tf
    - variables.tf

In this case, you may want to just copy and paste the Terraform code which is completely fine if you want to get this set-up quickly.

Remember that it is IMPORTANT not to hard-code any credentials here

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 2.70.0"
    }
  }
}

provider "aws" {
    profile = "default"
  region = var.aws_region
}

module "website" {
  source = "./.deploy/terraform/static-site"
  domain_name = var.domain_name
}

The module block links to all the resources that make up our static website.

variable "aws_region" {
  type = string
    default = "eu-west-1"
}

variable "domain_name" {
  type = string
}

Whilst we are here, let’s create a simple HTML website that also has a link to an “About Me” page.

<!DOCTYPE html>
<head>Welcome to my static website!</head>
<a href="/pages/about.html" >About me</a>

<p>This site was deployed via cool Terraform magic</p>
</html>

<html>
<h1>About Me</h1>
<p>I'm a page!</p>
</html>

Setting up your S3 Bucket

s3.tf

Creates a bucket that corresponds to the domain name you input (linking back to the variables.tf file)
Adds a policy to the bucket and has an ACL that is set to public-read (so that anyone with the link can access it)
With the website {} block, this S3 bucket now knows that, “Oh, I’m going to be used as a website! Yay!” And maps the index_document in the root folder to index.html as well as the error_document (i.e. what happens when a user goes to your-website.com/random-made-up-page)

resource "aws_s3_bucket" "my-website" {
  bucket = "${var.domain_name}"
  acl = "public-read"
  policy = data.aws_iam_policy_document.my-website_policy.json
  website {
    index_document = "index.html"
    error_document = "error.html"
  }
}

s3-iam.tf

This creates the policy for your S3 bucket. You can read up on AWS policies here.

data "aws_iam_policy_document" "my-website_policy" {
  statement {
    actions = [
      "s3:GetObject"
    ]
    principals {
      identifiers = ["*"]
      type = "AWS"
    }
    resources = [
      "arn:aws:s3:::${var.domain_name}*"
    ]
  }
}

Setting up Route53

route53.tf

What could this Terraform code be creating, I wonder? 🤔 That’s right! A Route53 resource! Again, if you’re familiar with navigating the AWS Management Console, some of these fields may look familiar. In this case, we’re creating an Alias record that corresponds to the S3 generated URL from the bucket we created.
Bear in mind - you do need to purchase a domain for this to work.
Notice how I’ve used the domain name variable here again.

resource "aws_route53_record" "www" {
  name = "${var.domain_name}"
  type = "A"
  alias {
    name =  aws_s3_bucket.my-website_bucket.website_domain
    zone_id = aws_s3_bucket.my-website_bucket.hosted_zone_id
    evaluate_target_health = false
  }
}

Running a Terraform plan / apply

Now for the fun part! Let’s start initiating the Terraform and see all the code that you just wrote in action.

Some of key Terraform commands:

terraform init
This initiates the Terraform based on the config in terraform.tf

terraform apply
If you’re all happy with what you see in the plan then go ahead and run an apply. This creates all the resources on AWS.

On your AWS Console, you should see all your resources built! Magic, right?

As easy as it was to create your infrastructure, you can also delete it all at once too with…

terraform destroy

🥳🎉🍾 Powerful, right? That is Infrastructure as Code for you! 🎉🍾🥳

Using Jenkins - some pointers

I won’t dive into groovy in this blog post (there’s already quite a bit of Terraform here to get your head around) but here are some pointers to think about:

Parameterise the domain name so that it takes in input from users
A simple bash script that uses the AWS S3 Cli to run an AWS S3 sync
Or if your index.html or application build is not committed to a repo (e.g. GitHub or BitBucket) then potentially a way to upload and unzip your application build onto S3 via a File Upload parameter.

Other things to think about

The solution above doesn’t cover using HTTPS, but it’s wise to do so. You could add Cloudfront to front your S3 website! Give it a go. 😀

Let me know if this was useful to you, happy playing around in da Clouds!

☁️ How I use AWS S3 to host images on my WP blog

Pauline P. Narvas — Sun, 09 May 2021 11:39:39 +0000

As I learned more about Amazon Web Services (AWS) at work, I was curious to see how I could make use of their services for my own stuff. I’ve seen the power it has in big enterprises and was keen to leverage whatever I could with my smaller projects.

Wordpress

I’ve used WordPress for most of my blogging life. Although it does has it’s faults sometimes (feeling “clunky” being the most annoying thing for me), I’ve always returned to it because it just serves my purposes of this blog – to blog.

When I launched into the Pawlean 2.0 rebuild, switching from building my themes in PHP to using Next.js and the WP API, I was spoilt by the performance improvements! Since then, I’ve been keen to look at other ways that I can optimise my traditionally clunky blog.

The media Library

The media library is where all your images, videos, pdf files, audio files, etc. reside. Essentially, it is the place where you upload all the different types of media files to use on your WordPress blog. This is usually stored in the disk. If you have access to cPanel or even direct ssh access, you can see this for yourself under the folder “wp-content“.

If you blog for a long time, you can see how this folder can start to build up with files over time. This results in needing extra disk space and can contribute to slow performance.

I’ve been blogging for over 10 years, with Pawlean.com online since 2015. With most of my posts heavy with media, my blog’s performance wasn’t doing too well. I was keen to look for a different solution!

AWS Simple Storage Service (S3)

“Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface.”

I always like to think of S3 as a place where you can store files (objects.) Similar to Dropbox or Google Drive, but with the ability to use the files in any way that you like including embedding onto your posts. 🖼

The benefits of S3 are endless. My favourites are it’s super cheap to use (I pay £1 every month to host all the images on my blog), it’s flexible, it’s scalable and has been designed for 99.99% availability which means it is almost always available. You can read more about S3 and it’s benefits here.

There’s several ways that you integrate the use of S3 onto your blog. My thought process at the time of implementation of this was:

I keep the existing media files as they are in my WordPress server, and just use S3 moving forward – uploading to the console, setting the objects to public and copying the image URL. This was the easiest way, but was too manual for me.

I somehow migrate all the media files across all my posts, replace all the image URLs to S3 URLs and moving forward, every time I upload an image to my blog, it automatically uploads to S3. I wouldn’t have to log into the AWS Management Console and manually upload images to S3 to then embed.

Although 1) was tempting as it was the easiest route, I decided to do 2) because it felt like the best way to improve my blog’s performance in the long run as well.

Offloading media from your WordPress server to AWS S3

1) Log into your WordPress Dashboard. Head over to Plugins > Add New.

2) Install these two plugins:

Amazon Web Services – Includes the Amazon Web Services PHP libraries, stores access keys, and allows other plugins to hook into it.
WP Offload Media Lite – Automatically copies media uploads to Amazon S3, DigitalOcean Spaces or Google Cloud Storage for storage and delivery. Optionally configure Amazon CloudFront or another CDN for even faster delivery.

3) In the Amazon Web Services plugin, add in your Access Key ID and Secret Access Key. If you’re unsure where to get these, check out this tutorial. It's recommend that you DO NOT use your root AWS account. Create a new account with limited permissions to S3!

4) Assuming that you’ve successfully linked your AWS account and created an S3 bucket with the correct permissions, this should be relatively straight forward. In WP Offload Media, you’ll find a bunch of different settings to go through.

My settings have been set up so that when I upload files to the Media Library (found in wp-content/uploads), they copy it to the bucket I specified. When I upload a file, by default, Year/Month is added to the image URL.

Rewrite Media URLs does what it says on the tin! It rewrites all the existing image URLs embedded on your blog to the URLs generated by S3.

Remember to tick this last option! This ensures that when you upload new files to your blog, it gets rid of a temporary copy in your WordPress server. This is actually how this plugin works: you upload a new image > this is uploaded to the server > this is copied to S3.

If you don’t tick this last option, it keeps that temp copy onto your server, which is not what we want for better performance!

5) Once you save your settings, it should start kicking into action! Wait around for a few minutes because it can take a while depending on how many images are on your blog.

Head over to your AWS Management Console > S3 and in the bucket you specified, you should start to see images being uploaded by Year/Month.

Images that are uploaded, are also automatically get different sizes which can be used for things like thumbnails or featured images for posts. Pretty neat, huh?

You’re done! 🥳 If you ever need to start from scratch or move your blog at all, your images will be safely stored away in S3 which makes migrations less stressful. Win.

If you get stuck at any point, please refer to the plugin’s documentation which goes into more detail and answers to some frequently asked questions.

Go and enjoy your faster blog, now backed with a highly scalable, highly available, detached media library. 😄