Forem: StackPulse

Reliability as an Inseparable Part of Software Engineering

Leonid Belkind — Tue, 19 Jan 2021 13:25:38 +0000

The world is changing:
I feel it in the water,
I feel it in the earth,
and I smell it in the air.
J.R.R. Tolkien, The Fellowship of the Ring

For someone who has been a Software Engineer, building enterprise software, during the past 20+ years, the changes our world is undergoing are as clear and evident as they were for Lady Galadriel of Lothlórien in the epic by J.R.R Tolkien. The meaning of “developing and delivering a software product” as we saw it in the 1990s, when I began my professional career as a Software Engineer, and the meaning of the same thing today are very different.

The differences start with the definition of a “software product”. Traditionally, software vendors would be responsible for developing software and packaging it in a form that allows their customers (businesses or consumers) to deploy it on their own IT infrastructure and use it. This paradigm, also known as “on premises” software products is now being ousted in many B2B and B2C scenarios in favor of delivering “software services”. The distinction between delivering an on-premises product and delivering (potentially the same software as) a service is not just about the form of delivery or consumption, it changes the core set of expectations between the customer and the provider / vendor.

When a customer is acquiring a “service”, the expectations are stretching far beyond its functional capabilities into the domain of availability. The customer rightfully expects that the service is available for use with a defined level of user experience at all times (according to the Service Level Agreement with the provider). The shared responsibility model that was in place for on-premises products, where the vendors had to provide working software products, but the customers had to deploy and operate them according to vendors’ specifications has shifted heavily into the vendors’ side.

As per normal industrial evolution, both the customers and the vendors are supposed to benefit from this new state. Both of them, however, have new challenges to resolve (or new realities to adapt to) in order to be able to reap the benefits:

Software Vendors (turning into Providers of Software Services) need to adopt a new set of responsibilities related to operationalizing the services and delivering guarantees on the service availability
Customers of Software Services need to be able to verify (to the extent possible) that the service providers are operating within the expectations in terms of service quality (availability, responsiveness, etc…)

For providers of software services, Site Reliability Engineering is the discipline that allows adopting this new set of responsibilities, guaranteeing their execution and proving that their services are delivered reliably. Delivering a reliable service and proving its reliability are two very different notions.

In the past, traditional IT operations have ensured reliability of their assets and infrastructure by introducing and following strict operational procedures. Those procedures were applied to “complete products” that were delivered to the IT organizations for operation. The operators could influence the infrastructure, the configuration and the procedures related to the product, but not the product itself. This limitation has clearly introduced a cap on the effectiveness of operations that could be delivered by such organizations.

Site Reliability Engineering is changing the rules of the game by removing this “glass ceiling” and allowing the reliability considerations to influence the product all the way from its architecture and design, through various stages of implementation and testing all the way to its deployment in production. As this approach is being adopted by a growing number of of organizations, it is becoming increasingly clear that this is not just a dedicated “profession” (carried out by dedicated engineers called SREs - Site Reliability Engineers), but a pillar in Software Engineering in general, similar to aspects such as, but not limited to Software Architecture, User Experience, Automated Testing, etc…

Just as the DevOps philosophy has merged the domains of Software Development and Software (Service) Operations, Site Reliability Engineering is relying on this approach to define reliability challenges not as operational ones (where regulations, processes and escalation procedures would be way of delivering results), but as engineering ones - where a set of mechanisms engineered into the product would guarantee the desired results.

Reliability engineering, therefore, is a notion that should affect every stage of software development:

Architecture and Design with reliability metrics and goals in mind
Implementation that ensures reliable operations when deployed to production
Testing that introduces verifications and gates focusing on reliability and not just functionality
Deployment that ensure reliable roll-out Monitoring surfacing potential reliability issues ahead of time (or, at least, when they, unfortunately, take place)
Operationalization that allows both ensuring reliable functioning of the service and provides continuous feedback loop back to Design and Implementation stages to improve the reliability of future product iterations

Building a Reliability Platform the way we do it in StackPulse means, first and foremost, that we have to adopt the above principles as the foundation for our Software Development Lifecycle. We firmly believe that this is the only way to ensure that the users of our services are guaranteed to have the best and the most reliable product at their disposal.

This also means that the platform we are delivering to our customers allows them to adopt a similar state of mind and to build mechanisms that will drive the reliability aspects in their respective software architectures and development processes. Becoming an integral part of the SDLC means being able to operate within the main principles upon which the lifecycle is based, such as:

Single “Source of Truth” is based on the software version control repositories
“Fail fast” verification of all changes prior to release to production
GitOps flow initiating and managing the progressive deployment of changes released to production
Full transparency and metrics are built in to each process for detailed telemetry

Additionally, we believe that the best ideas are not coming from a single source, but are a result of a collaborative effort. That is why our platform allows developers to share the mechanisms they build for reliability, to open them to community feedback and contributions and to make the world a more reliable place as a result of a joint effort. We firmly believe that together we can make a real difference in the reliability of different software services in the world, by sharing our challenges and our practices. We see ourselves as an enabler of such discussions and as a facilitator of knowledge sharing in the field.

The platform that we are about to launch consists of the following components:

Alert Enrichment & Analysis - To reduce alert noise and fatigue with real-time enrichment, triage, and analysis
Automated Incident Response - Automated, code-based playbooks to investigate and remediate production incidents; reducing toil, reducing MTTR and meeting SLOs
Incident Lifecycle Management - To streamline communications, speed remediation, and automate data collection during incidents

This is just the beginning. Our mission is clear: we are delivering a holistic platform that allows engineering organizations to interleave reliability into all stages of their software development and operations. Doing so has to be a continuous agile process, constantly evolving and relying on data-driven insights. This is the way we are doing it ourselves, and this is the way we would like to do this together with our users.

We are StackPulse! Join us in making the world a more reliable place, one software service at a time!

⁉ Why I started developing 💡 my new software project by building a 🚀 Continuous Deployment 🔃 pipeline

Leonid Belkind — Sun, 01 Mar 2020 15:41:15 +0000

Somehow, in recent years, when discussing software delivery pipelines in modern companies, people would casually drop the term CI / CD (which means Continuous Integration and Continuous Delivery / Deployment) always combining both practices. In many cases it is implied that the two practices always go together, or, maybe Continuous Delivery follows Continuous Integration as an optional step.

During recent years I had an opportunity to oversee the establishment of a software delivery pipeline twice, implementing only Continuous Integration first time, and starting with Continuous Delivery in my latest project.

🙈 TL;DR / Conclusion

There are HUGE differences between the two practices. Differences that don't just have impact on the technology used, but also on the culture of the engineering team. Implementing Continuous Integration only and hoping that it is a sufficient foundation to switch to Continuous Delivery in the future is a myth.

👊 Which is which and why do you need them?

To explain the choices that we made, one needs to understand the difference between CI and CD. For those who feel confident that they understand it, skipping the below paragraph is recommended. For those who are suddenly not so sure - read on...

☝ Continuous Integration

Continuous Integration is a practice targeted at maintaining a working "integrated" system consisting of multiple components being worked on in parallel by multiple engineers / engineering teams.

The following diagram demonstrates a simplified Continuous Integration process:

Each of the developers involved is delivering a change in one of the components. The CI constructs an environment consisting of the latest components with the new addition and performs a cycle of automated tests.

The need to implement a Continuous Integration in businesses that depend on their ability to deliver software efficiently and reliably is very obvious. The alternative is straight out painful. Doing periodic "big bang" integrations, bringing together development projects that were run separately and hoping that they would magically run together - it's a huge challenge.

I remember the old days where building a multi-component software server in a large R&D organization would result in a very painful iterative integration time period, during which tons of builds would be delivered to QA and would be pronounced "dead on arrival".

Quite frankly, I cannot think of any good reason why anyone in this day an age would prefer not to use Continuous Integration. Sure, building it effectively would require investing in automated testing, which is yet another thing that just should be done without any reservations.

The only reasonable situation in which CI shouldn't be used, is the case where the organization maintains a significant amount of legacy software components that cannot be validated with automated testing. If this is the case, then, unfortunately, pretty much everything in this article won't apply. Trying to do Continuous Integration without sufficiently reliable automated testing for components is going to generate something of a very little practical value.

✌ Continuous Delivery / Deployment

Continuous Delivery and Continuous Deployment are terms that quite often get used interchangeably. There is a difference between the canonical definitions of the two which is important to note, in order to user the correct term:

Continuous Delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, doing so manually. It aims at building, testing, and releasing software with greater speed and frequency.
Continuous Deployment is an approach, where the software, after undergoing a set of automated reliability gates, actually gets deployed in the production environment in a fully automated (and controlled) manner.

While Continuous Delivery may, indeed, be a natural extension to a well-implemented Continuous Integration, Continuous Deployment requires a significant amount of investment, both technological and methodological, in order to implement successfully.

The above diagram depicts the full flow of a Continuous Delivery cycle implemented with Canary Progressive Delivery (one of the possible strategies for a responsible automated update of software components in production environments).

👟 Implementing just CI

In order to implement Continuous Integration successfully, an organization would need to invest in a number of technical infrastructural components, as well as to encourage (or even enforce) certain rules on the "Definition of Done" for Software Developers.

Three main components listed below allowed us to enable efficient CI and gain a consensus (among the members of our engineering team) that it is a valuable asset for the organization.

🤖 Automated Tests

An absolute must for any CI environment, automated testing of your code provides the foundation for an ability to claim that a code change has been successfully integrated.

A lot has been said about automated testing methodologies. At this point, it is enough to state that testing automation should be introduced on multiple levels (and not just either Unit Testing or End-to-End Testing). Layers, such as Module Testing, Contract Testing, Performance Testing and even Automated Security Tests should be considered, depending on the requirements of your project.

It was our experience, that introducing all relevant tests incrementally for every code change and treating these as an inseparable part of the development cycle (rather than separate tasks) ensured that our ability to successfully gate software changes in a dynamic environment keeps the overall system stable.

🌀 CI Pipeline

During the early days of Continuous Integration, when it was still called Software Configuration Management Automation, organizations used to automate software building, testing and packaging tasks writing custom scripts / code. Jenkins was, probably, the first CI and later also CD pipeline tool that has emerged as the main-stream-adopted (almost de-facto standard) for building Automated CI Pipelines.

Additional products, both self-hosted open source ones, as well as SaaS-delivered (Travis CI, Circle CI, Gitlab Pipelines, GitHub Actions, Google Cloud Build and more) have since taken the architecture and the instrumentation of CI lightyears ahead.

📦 Artifacts Repository

A very important component in the pipeline, repository for storing the tested / verified "deliverables". Artifacts Repositories have evolved from a file-system with a predefined structure to a more sophisticated software solutions, capable of delivering the "artifacts" using various Software Package Management / Distribution systems.

👞 Implementing CD on top of CI

Do we need to implement Continuous Integration first, and then consider working on Continuous Delivery? A great article on the subject suggests, that, while this is a conventional approach, it can, actually be done in a different order (with some unorthodox thinking). While this is a valid opinion, all of the Continuous Delivery environments familiar to the author of this article are, actually, built on top of Continuous Integration pipelines.

On top of the components listed above, following ones will need to be added in order to implement Continuous Deployment. Above all, the understanding that every delivery will impact real users of your service is something that needs to be generally acceptable by your engineering team. Engineers should feel confident in delivering changes, knowing that the deployment is done in a responsible manner and that the system can be observed properly to make sure that it functions according to expectations.

👏 Operator / Trigger for Deployments

This is, in fact, the only 100% mandatory component for implementing Continuous Deployment. This is what ensures its continuousness. The goal of this component is to trigger deployment process automatically upon either a completion of a Continuous Integration for a certain change, or upon any other automated trigger.

There are multiple ways to implement such a trigger, both including custom-developed code and open-source projects, such as Flux CD or Spinnaker.

🙌 Progressive Delivery Controller

Progressive Delivery is a methodology that ensures gradual adoption of newly delivered components by some users of the overall system (but not by everyone at once). Coupled with Health Metrics Monitoring, it can be used for a fully automated responsible roll-out.

Depending on the nature of your service, using Progressive Delivery is not a must. If you can replace versions of various components of your software in the production environments without worrying about its impact on the end-users, you don't need to do anything special.

If causing interruptions to current users of your service is a concern, then some sort of a progressive delivery controller is required. Open Source projects, such as Argo Rollouts and Weaveworks Flagger deliver this functionality for Kubernetes environments.

There are various methods for performing Progressive Delivery, such as, but not limited to Blue-Green or Canary upgrade.

👈👉 Traffic Split Manager / Load Balancer

Since at various stages of a Progressive Delivery process both the "Stable" and the "New" versions of the same component will exist simultaneously, something needs to be able to balance traffic between these two. Most Progressive Delivery Controllers are capable of controlling the traffic split management by interacting with notions, such as Load Balancers, Service Mesh controllers, ingress controllers and other components capable of directing North-South, as well as East-West traffic to various instances of destinations.

👍👎 Health Metrics Monitoring

The ability to establish whether your component is "healthy", or whether it is functioning according to expectations, is a mandatory part being able to do automated deployment. Without this ability, your developers are "blind", not knowing whether the system is functioning or not.

The monitoring can be done either using passive methodology (i.e., your component emits various metrics or logs and something monitors whether their data is reflecting a properly functioning system) or using active methodology (periodic probes/sensors are executed against the system, verifying that its response is according to what is expected).

When considering systems for collecting metrics, Prometheus rules supremely as the most widely used open source tool, easily integrated with all the leading Progressive Delivery Controllers.

💫 Tying it all together

Having established what Continuous Integration and Continuous Deployment are, you can now judge the requirements for your specific case. The article above has focused on the technical components that are required to achieve either of these capabilities, mentioning that the engineering teams and their culture are also an important factor.

Deciding on the proper strategy for your software service is up to you. One important piece of advice is not to underestimate the challenge of changing the direction in the middle and its impact not just on the technical debt, but on the engineering team.

Choose wisely...

P.S., if you are interested in how we implemented an end-to-end Canary Continuous Deployment, refer to this excellent blog by Or Elimelech.

👀 Building Developer-First Products with Developer Experience 📒 (DX) 101 🙇

Leonid Belkind — Sun, 23 Feb 2020 12:14:07 +0000

With multiple trends pointing at the growth of the developers buying power in organizations, the market for developer-first products and services is under a constant growth. Companies like NewRelic (IPO in December 2014), Atlassian (IPO in December 2015), Twilio (IPO in June 2016) and other more recent successes are demonstrating that delivering a product for developers can grow to a significant business.

Product design and its value/benefits for the end-users are becoming a much bigger part in the product (and, therefore, business) success (💰💰), since it's the end-users (i.e. developers) that get to decide what to buy. In fact, a modern goto-market approach based on the Product-Led Growth principles is focusing on the importance of User Experience in the buying decision.

In this article you will find:

General approach based on involvement phase of a user with the product
Aspects to consider when attracting prospects' attention
Aspects to consider when offering a product trial
Aspects to consider for daily use of a product
Aspects to consider allowing users to amplify the product impact
Summary and recommendations

💡 Approach based on Interaction Phases

Based on the above, the goal of Developer Experience design is to match the needs of developers (users in our context) in the best possible way. Do these needs remain the same throughout the lifecycle of the interaction of a prospect developer with the solution or do they change? According to observations made by a number of vendors in the past, the needs / expectations of the users are very closely related to the phase of their interaction with the product.

Lets consider breaking the product interaction lifecycle into four basic stages:

Notice / Attract - How does a prospect notice the product and gets attracted to the value it may introduce
Try - How convenient is it to try the product and start getting the value
Use - What is the experience provided by the product in the regular ramped-up day-to-day use
Amplify / Grow - What amplifications does the product offer for those who want to leverage it beyond the typical use-cases

For example, functionality related to ease of onboarding and out-of-the-box settings would be attributed to the try stage, whereas advanced integrations with additional products or analytic reports would fall somewhere between use and amplify/grow. This phase-based approach is also applicable to the more traditional UX, but, when dealing with DX, it can help prioritize and delivery systematic impact on the journey that developers have with the product.

The guide below provides suggestions on particular topics to take into consideration when designing a developer-first product targeting each of the interaction stages mentioned above.

🕛 Notice / Attract

How will the developers (that are potential users of your product) learn about its existence? Where will they generally look for solutions for the kind of problem you are aiming to solve? Will your product be somehow represented there and will it stand out from the "noise"?

Web site

A web site representing a product/service targeting developers should contain fundamentally different information, when comparing it to a typical B2B product site. A lot has been said about the content and the language that developers appreciate. Clarity, brevity, being to the point, using the proper terminology - all these are among the leading factors appreciated by the target audience.

When developers are your target audience, you need to consider the state of mind involved. People who don't appreciate being sold to. People with limited time and attention span, who'd like to understand the value proposition and the "how to" as soon as possible.

Here is an example of a homepage targeted at such an audience:

Notice the very sharp difference between this and the following homepage, that is targeting an enterprise decision-makers audience:

Both the substance and the language (not to mention the look-n-feel) are very much affected by the audience being targeted.

Community presence

Developers make intensive use of various communities to look for solutions for common challenges. Depending on the domain of the problem, Stack Overflow, Dev.to, Reddit or other community information exchange platforms are often serving as the first place where one would expect to find some sort of information about potential solutions to use.

Here is a sample search result for finding an API to send SMS on StackOverflow. Naturally, developers facing this problem will be inclined to try solutions / approaches suggested by their colleagues.

Open Source presence

Releasing useful open source projects is always a huge contributor to building the developer-facing presence. Even when your product is not an open source, having open source repositories for your SDK, samples and additional resources can be considered a huge plus.

Here's an example of an open source module representing an SDK for a proprietary service. Note the amount of stars on the Github repository.

Integrations / marketplace

Will your solution be used by its target users (developers) in integration with another developer solution that is already in their stack? Is that other solution extensible? If so, one of the ways to improve the overall experience is to publish a plugin.

An example of Service Providers publishing HashiCorp Terraform Plugins.

Documentation

While nobody is reading the documentation thoroughly during this stage, checking that it exists and that it conforms with expectations / best practices can definitely overcome certain concerns.

Stripe API documentation, is an example of a very professionally built developer-facing documentation.

🕑 Try

Let's say that we 'aced' the first step and managed to attract the attention of a potential user. What does it take to try our product (remember: developers like to get their hands 'dirty' and try things)?

Trial of the product can be done in a number of phases. Some people would be so convinced by the product value (from the previous stage),that they'd be looking for trying out the product in their environment. Others - would like to take a more conservative approach and would be looking for some playground not linked to their environment.

Online playgrounds

Online playgrounds, gaining traction recently, are solutions, integrated in the product/service website, that allow trying it out (getting a hands-on experience) without deploying anything on one's laptop/cloud.

One of possible implementations of the online playground is via integration into a tutorial / documentation, similarly to how it is done in the Golang Tutorial. Another great example is a Kotlin Playground by JetBrains.

Playgrounds can cover either a single process/tool, or represent a whole virtualized environment, such as ones created with Katakoda.

Delivering an effective online playground may reduces the upfront costs required for trying out your solution, and, if the solution itself provides the relevant/expected benefits, heavily increases the chances of progressing ahead from the trial stage.

Containerized trial

If your product requires deploying any agents or servers in the customers' environments, providing a containerized trial (even if it is not a fully production-grade supported deployment) will ease on the initial deployment and will reduce the friction. Using Dockerhub for publishing such container images with easy documentation on how to deploy them makes the trial deployment much more user-friendly.

For some of the examples (that may also be used for production-grade containerized deployment), reference the MySQL Server, TeamCity Server or NetData Dockerhub postings.

Free / Trial Service Account

When your product/service contains a service (either SaaS Management or a full service) offering, it is 100% imperative to offer either free or trial service accounts.

Time-limited trials used to be more acceptable option in the past for both consumer and enterprise services. In modern developer-first products, this approach is a bit more problematic, and functionality/scale-restricted or even freemium offerings seem to gain more traction.

Almost every time developers will prefer the free/freemium offers, due to the following:

There is no immediate "sales" motion behind them. Free/freemium versions are there to be used as real tools, as long as your demands fit the constraints.
Using (and benefitting from) the free version increases the "stickiness" of the solution, as it can be used in production. This, in turn, increases the chances of converting the account to a paid one, as it is difficult to do a meaningful trial with strict time limits. (Even psychologically, investing in such a trial, knowing that it might be a complete throw-away is something that is less optimal)
Depending on the value your solution provides, it can be used for personal/open source projects in its free mode, increasing the loyalty and the possibility that it will become adopted in work environments.

Defining the conversion point for freemium accounts to become paid is a science (or, some will say 'art') by its own and deserves a dedicated discussion / guide. Some of the most prominent approaches are:

Consumption based (size of storage, amount of events)
Users based (limited amount of users in free/freemium accounts)
Functionality based (collaboration features, enterprise features, etc... are only available on paid accounts).

CLI / API Samples

When developers interact with products/services, some of the user flows rely on Command-Line Interfaces (CLIs) and Application Programming Interfaces (APIs) rather than User Interfaces. This dictates that both CLIs and APIs should be treated as the face of your product and should be designed, tested and documented with the same attention to details as UI.
This approach can be extended to things, such as, but not limited to, Chatbot interface, if your product is offering one.

Providing a documentation in a developer-friendly format, or even providing open-source samples of usage of your CLI/API on Github can go a long way in terms of encouraging the adoption of your technology and making its frictionless.

Out of the box settings

Last, but definitely not least, on the trial stage is the ability of the product to function and to provide actual value without someone needing to spend time and other resources to configure it. In fact, as mundane as this point may sound, it is a real difference maker when it comes to the trial stage.
The faster the product starts showing real value, the more likely the probability that its prospect (the person trying it out) will consider adopting it.

In recent years, the industry has even come up with the measurement of this, adopting a term of Time to Value. The investment, in the trial stage, is that of our prospects' time and the value the product shows is the expected return on that investment.

🕒 Use

This is a critical stage: our potential users arrive with the maximal 'inflated expectations' to this point, based on their successful trial, and are starting to use the product in their environment. The ability to be as good in 'real life' as in demonstration/trial is crucial in successful product adoption and in achieving customer loyalty.

User Interface

If the User Interface plays a significant role in the interaction of various users with the product, its design is a very important aspect of the overall product work.

There are various approaches to designing user interfaces and a lot has been said and written about this.

When dealing with developer audiences, it is important to remember that the UI will serve as a tool for completing tasks (more than in any other enterprise product). Special emphasis should be made on what is the role of the user interacting with the system, which tasks the user is looking to accomplish and which data/information exists in the system to help them accomplish the tasks.

CLI / Scripting

Automating interactions with developer-first solutions is a significant portion of user-flows for developers.

Similarly to the importance of CLI in trying out the product, its importance in adopting the product for everyday use is also very significant.

When designing the CLI syntax, it is always recommended to stick to standards (such as POSIX CLI Syntax or Microsoft's), reducing the adoption friction.

Documenting how to do this, providing samples and knowledge base article around automation via CLI will be of a great value to the user audiences.

API

Just as with CLI automation, providing integration and automation capabilities via APIs increases the value your users will extract from the product. Even more so, it can have a positive effect on the stickiness of the solution - once it is integrated into automation processes, the extent of its usage grows without a dependency on human users and their availability.

Designing an API, much like designing a User Interface, has its own methodologies and best practices that should be studied closely and applied, in order to increase the chances of success.

Customization of Settings

It is very likely that during the usage stage of your product its users will want to customize its behavior. Out-of-the-box settings are only useful for a handful of common scenarios, and the ability of the product to provide a flexible configuration is a big plus.

In fact, the experience learnt from years of designing Enterprise Security Products have delivered a methodology of Policy Model. In these products (the methodology can be extended to other products as well), the main value is the ability to enforce an organizational security policy using the capabilities of the product. The policy, in turn, can be customized either for different networks, different users, different hosts, etc...

The level of customization (on a scale of flexibility=complexity and lack of flexibility=constrains) should differ per product. It is very difficult to come up with rules of thumb in this department, except for a very clear need to do a meticulous system analysis in this case.

Configuration Management

Regardless to complexity vs simplicity of the chosen configuration/policy model, the question of a management lifecycle for configuration (Configuration Management) is always important, especially to developers. Every change in the product configuration that can affect production environment could/should undergo stages of review, delivery and monitoring.

Once upon a time, many mature/advanced products offered some sort of a configuration management lifecycle built-in. Some products had a two-phase commit of changes, others added versioning and approval flow.

Developers don't need any of it. They already have configuration management flows centered around software versions repositories (such as, for example, GitOps.
Automating configuration management in your product via CLI or API, as mentioned above, allows developers to use their existing configuration management lifecycle and just to a the configuration of your product to it.

Providing integration with configuration automation tools (CAPS - Chef, Ansible, Puppet, SaltStack), or, for more cloud-native-adopting users - HashiCorp Terraform Infrastructure-as-Code could make your product fit very seamlessly into existing configuration lifecycle. (There are significant differences between mostly imperative configuration management offered by CAPS tools vs declarative one by Terraform, but this is a subject for another article.

Documentation

Developers expect the product they are using to be documented in a form that looks familiar, provides answers to expected questions, and doesn't overburden with unnecessary formalities.

Certain types of documentation have best practices or even a golden standard (sometimes, even tools that help creating it). For example, when documenting REST APIs / Swagger files, consider using ReDoc. As the industry providing the APIs has undergone a significant maturity growth in recent years, best practices have been developed and should be adopted.

Similarly, when considering documenting CLIs, best practices by Google or Microsoft, as well as open source standards, like docopt, can provide a good direction.

Being standard, looking familiar - will eventually contribute to reducing the friction of adoption.

🕔 Amplify / Grow

While every product / solution is targeting solving a specific set of problem(s), exceptional solutions can deliver a longer-lasting (and, therefore, much more "sticky") value to their users.

Developers will benefit from long-lasting value, allowing deeper integration into their stack, ability to handle a more diverse set of use-cases, providing insights and data beyond what was previously available, etc...

SDKs

SDKs are the next step of ensuring that your users "stick" to your solution for a longer term, after automation via CLIs and APIs. SDKs usually allow a richer experience of interacting with your product and ensure closer ties between the developers and the "platform".

When providing SDKs, it is important to ensure that they are available for all the relevant programming languages (depending on the type of the audience), have a great developer-friendly documentation and come with usage samples.

One critical thing is to have them always up-to-date. This is among the things that ensure the trust of the developer using the SDK in the platform. If the SDK is not up-to-date and there is a functionality drift between the product capabilities and what is available / accessible through the SDK, the willingness to trust it and to use it will inevitably go down.

Integration with additional solutions

In the environment that desires to automate everything that can be automated and gain maximal possible value from collected information, the most important currency is the ability to integrate with other elements of existing pipelines/workflows.

While having an open and well-documented API allows integration with practically anything, it also requires effort to implement. Providing out-of-the-box integrations (sometimes also called "single-click integrations") with products/solutions that are frequently used by the same people that use your product, is a clear win. Contributes to user satisfaction, provides value beyond the core values of your product and increases its stickiness.

☑ Summary

The above guide was built in an effort to drive the attention of individuals building developer-first products to various product aspects and to try and map the importance of each, as the user progresses in product adoption.

As developers are becoming a more significant force in choosing the organization's technological stack, the amount of possible options will continue growing. Feel free to add your experiences in designing developer-first products as comments.

♨ Strategies for investment in 🔩 Tech Debt vs 💸 Product Debt when building new 📦 software products

Leonid Belkind — Tue, 11 Feb 2020 12:57:21 +0000

Spaghetti code, architecture that isn't well-thought-out, growth and supportability challenges - being a software engineer, you get plenty of opportunities to experience these and similar effects throughout your career. As much as we love to criticize others for choices they made (and that we have to live with), let's be serious for a moment: Very few engineers are actually making a conscious choice to write a bad code.

"I have an idea! Lets write a fundamentally bad code and make the life of everyone who ends up using it and maintaining it really miserable!"

This is not how it works. Behind every "bad" software module that someone has to maintain or use, there is always someone who either really didn't know any better or thought that implementing it the way they chose seemed to be a good idea at the time.

As the field of software engineering evolves and becomes more modern, knowledge gaps reduce, and, while not everyone has to be the mean lean coding machine, there is a lot of available knowledge and support that would explain proper ways to implement certain software solutions. Even if you are inexperienced in what you're doing - "not knowing any better" becomes harder and harder in our always connected, Google this/Google that world.

So, in a nutshell, it all boils down to choosing the right strategy of managing investments (and, therefore, progress) in the proper infrastructure / architecture vs implementing new product functionality. Choosing the right strategy is akin to implementing a design pattern. Upon reaching this conclusion, one can start considering various strategies based on the needs and the constraints of the business requirements. Below we are listing a number of prominent strategies, talking about the pros and the cons of each.

1⃣ Build Product, Deal with Infrastructure / Architecture Later

This approach is rumored to have originated in the start-up world. Some people will blame The Lean Startup principles for promoting the idea (not necessarily accurate), while others will cite the general challenge younger generations have with Delayed Gratification. In real life, one can find many cases of this approach taking place in established enterprises, that are interested in innovating, while being reluctant to make significant investments prior to seeing any "justification" for a future ROI.

In many cases, the first product that is being developed is sometimes nothing more than a trial balloon, designed to prove or to disprove a certain thesis. The underlying logic is: why do we need to invest in the proper architecture / infrastructure if we don't know whether this product has any future at all?

While the logic of the above assumption is correct, and, one doesn't really progress with validating her/his product-market-fit assumptions by investing in the infrastructure for the product - the problems arise when the code actually sticks around.

After all, unless explicitly designated, code that someone developed rarely gets thrown away completely. The solution may undergo a pivot, the company may change positioning or messaging for the product, but the technological investments may actually remain. Not to mention the case where the original thesis, or a certain part of it, gets actually proven and the company continues developing the idea.

What happens then? Your team (hopefully, a collection of capable individuals) has been running amok developing the product functionality, they have actually created something that looks and acts like the real deal, but its foundation is flaky.

It is very difficult to pause and make a re-factoring effort at the time when your product is starting to gain traction. After all - everyone expects the opposite to happen - you should floor the pedal in developing the features and conquer the world.

Below diagram explains (schematically) what can happen in this case:

The X Axis represents the timeline (with major project/product milestones, such as POC, Beta, MVP, First Full Release and Second Full Release) whereas the Y Axis represents the effort (and, implicitly, the progress) on either the product debt or the technical debt (Infrastructure / Architecture) fronts.

As described above, we start by investing most of our efforts and resources (that keep growing as our organization grows) on the product front, while convincing ourselves that we will gradually increase the infra investment as the product matures. By the time the product reaches its second significant release, we are expecting to be able to make a significant leap in proper technological foundation.

Sounds nice in theory, right? Let's layer the accumulating technical debt and the acceptable technical debt/quality on this graph to see:

We can clearly see the red line of the Accumulated Technical Debt climbing (why wouldn't it, when there is no sufficient investment) and crossing the dotted red line of the Acceptable Technical Debt (that, surely, doesn't have to be zero) at a certain point.

In this specific graph, this point is between Beta and MVP milestones. It is an example, of course, and, depending on the complexity of your technology and/or the technical acumen of your team, this point can shift to the right. Still, more often than not, this takes place and starts hurting your business at the stage where the business cannot afford this.

Even the optimistic part of this graph, showing that the technical debt starts decreasing after an effort was invested, is very questionable. Who knows if the organization will be able to support the cost of the investment, and who can estimate the impact of providing a product with a technical debt above the acceptable level?

In a nutshell, while all the proponents of this approach will claim that the beast is not as scary as it looks, the author of this article saw (by far) more companies/product teams that either failed, or got crippled by it, than those who actually managed the reap its illusive benefits.

Recommendation: Beware of Greeks bearing gifts. If choosing to go down this route, be very tight with your hand on the pulse of your technical debt.

2⃣ Throwaway POC, then Build the Real Product with Infrastructure

This approach sounds very sensible and easy to implement: all you do is define that your original code is going to have one purpose only - proving that the idea works - and then throw it away and write the real thing.

In theory, the approach works, if you have enough discipline to stick to it. Your investment in the lifelike POC will, hopefully, allow you to prove the product-market-fit, and then, armed with it, you can go and build the real product.

The investment / progress graph of such an approach may look like this:

The investment in the product features during the POC stage, hopefully, gives us the required confidence in the product-market fit. Then, we start developing the real product (this time investing first in the proper infrastructure and architecture, and then piling up the product features). We are further assuming that this work can be split between the Beta and the MVP milestones, delivering a properly-implemented MVP.

Lets layer the Technical Debt accumulation on top of this graph and we will receive the following:

The technical debt reaches unacceptable level quite quickly during the POC, which is fine by us as the POC gets discarded anyway. What happens after that (and the graph shows quite extreme assumptions) is that it is pushed down by the infra/architecture efforts during Beta and MVP stages and stays at the acceptable level all the way until the product evolves far enough from its original purpose - somewhere in the future. Again, depending on the complexity vs flexibility and volatility, the time axis could look quite different.

In theory - this approach makes everyone happy. In practice, in order to succeed, it requires a very adamant engineering team that will really throw away the prototype code and the business team that will understand the delay in Beta/MVP stages because of that. What is the alternative? Getting the same as our #1 approach, but with a higher technical debt (because the initial stages of the POC were developed under the assumption that it'd be thrown away, and here it is - sticking around).

Recommendation: Not all that glitters is gold. If this works, it is a solid strategy, but it is enough that one of the parties involved blinks and it can turn into a disaster.

3⃣ Infrastructure First, then Product, Continuous Refactoring

In recent years, over-simplified interpretation of The Agile Manifesto has motivated people to shun from making large up-front investments in software infrastructure prior to developing a working product. After all, if we should all embrace changing requirements (even late in the development), as well as having our highest priority to satisfy the customer, then investing in health and stability of our own infrastructure isn't necessarily the best thing to do. Delivering more customer-facing features is. Right?

Actually - wrong. Our customers, first and foremost, care about getting a stable, scalable and well-built product, and the fact that we expect requirement changes and will demand (from ourselves) to deliver response to these quickly, mandates that investing in a solid infrastructure and architecture is, actually, on a critical path to the main principles of the manifesto.

While, in theory, this delays the delivery of the first end-to-end POC to the potential customers, with a slightly larger (or more professional team) this could be mitigated. The graph of such an approach may look like the below:

While this graph also hides some assumptions, it shows the biggest desired benefit of the approach: the fact that the technical debt is constantly managed and, hopefully, doesn't cross the line of an acceptable debt in the foreseeable future.

Sure, skeptics can claim that this graphical representation is biased and that the approach hides potential of over-investing in the infrastructure and under-delivering on the product front. This isn't entirely incorrect. This approach is not a silver bullet and requires close management, just like anything else.

One important benefit that it introduces is that the technical debt is being managed (and kept at bay) from the very beginning of the product lifecycle. After all, keeping something at bay continuously vs allowing it to spike and then trying to calm it down is (arguably) going to be more cheap in the long run.

Introducing such an approach as a part of the organizational culture also cultivates pride in one's work and its deliveries. It doesn't exclude the possibility of a pragmatic decision to postpone certain infrastructure investments to future product milestones.

As previously, depending on the complexity of the solution, initial investment in the infrastructure (and the anticipated delay in functional delivery) can be higher or lower.

Recommendation: There's only one chance to start things the right way. Each situation may have its own right or wrong, but combining the desire to build a solid product with strong business connection and guidance, will, probably, provide the best results. There is no shame in admitting that not everything can always be done properly under all possible constraints and in choosing one's battles.