Forem: Sheroy Marker

Configuration Strategy for Continuous Delivery of Microservices

Sheroy Marker — Mon, 03 Jun 2019 18:18:45 +0000

This is the fifth post in the series - Continuous Delivery of Microservices. In the previous post, we talked about environment strategy - including artifact promotion and ways to leverage modern infrastructure for dynamic environments. In this post, we will discuss configuration strategy.

Introduction

An application’s configuration is everything that is variable across deployment environments such as development, test, and production. Deploying the same code but switching out certain aspects (like URLs to backing services, database connection information, credentials to third-party services, etc.) are examples of what I mean by variables in this context. Such configuration should be stored separately from the application code.

In a system based on a microservices architecture, configuration also needs to be distributed across multiple services. There are a couple of ways to manage configurations in a distributed way: *make configuration available in environment variables at deploy time, *use an external configuration server product designed to expose configuration

Here are three things you should consider for your microservices configuration strategy:

1: Manage application configurations centrally

An external configuration server is a more appropriate system for managing application configuration and introduces cleaner separation of concerns.

The configuration management code (in chef or puppet) can solely be responsible for cluster management. With Chef, updating application configuration would require a slow convergence operation of the cluster. With an external configuration server, updates to application configuration can be more dynamic without the need to update any other aspect of the infrastructure.

Another added advantage of this approach is that it forces consistent practices with organizing configuration by application, and environment.

There are a number of purpose built external configuration servers you could consider. The Spring cloud config server is a good option for Spring applications. With support for multiple backends, you could integrate with industry standard KV stores such as Consul for non-sensitive configuration, and Vault for sensitive configuration.

2: Standard process for distributing configuration

For microservices systems, it is possible to have different tech stacks across the systems. If one is handling configuration differently for different stacks, then the complexity becomes hard to manage. Therefore, regardless of the tech stack of a microservice, configurations should be distributed to nodes in a standard manner.

A technique we use is to supply configuration as environment variables per The Twelve-Factor App methodology. As a rule of thumb, always avoid distributing configuration files.

The Twelve-Factor app is a manifesto that provides some guidelines to be followed while building cloud-native applications. These guidelines let you build applications that are cloud-friendly. To truly harness the advantages of a cloud environment, an application needs to embrace cloud concepts such as elastic scalability, independently deployable and operable services, and statelessness.

3: Governance policy around secrets

Secrets such as API keys, passwords, and certificates need to be accessed securely. You need a governance process to ensure secrets access is managed appropriately. One technique we recommend to store all secrets is a central secrets store. The central external configuration server could provide this capability.

This central store gives you traceability on how and when policies were changed. That traceability goes a long way in setting up a governance process.

A tool we recommend to store secrets is the Vault by Hashicorp.

Example:

This is an example of an architecture where configurations are stored centrally in a config server and updated by the CD pipeline and pushed out to service instances.

At the top there is an abstraction of the CD pipeline. This updates the config server and then the configuration from the config server is pushed to the service instances. At run time, service instances are aware of how to consume this configuration. When setting-up an architecture like this, you need to consider how many configuration servers should you have. We recommend that you have one configuration server per CD environment, or at least one for production and one for all other environments.

Summary

This is the part 5 of our CD for Microservices blog series. We have talked about configuration strategy for your CD pipeline. In the next post, we will talk about the last consideration: remediation strategies for when something goes wrong.

Environment Strategy for Continuous Delivery of Microservices

Sheroy Marker — Mon, 13 May 2019 13:47:12 +0000

This is the fourth post in the series - CD of Microservices. In the previous post, we talked in depth about CI practices, in particular trunk based deployment and feature toggles. In this post, we’ll discuss environment strategy.

Environment plan

In some organizations people don’t realize the importance of an environment plan until they have too many environments and maintaining them becomes overwhelming.

An environment plan communicates the various environments that are involved in the path to production and their intended uses. It also communicates how your artifacts are promoted and the toggle states on these environments.

You can start with thinking about what environments need to be created upfront and talking about the intended use cases for these environments. Different groups in your organization will have different competing needs. Your environment plan should accommodate the needs from different parties.

Artifact promotion

Artifacts are one of many kinds of tangible by-products produced during the development of software. Some of these artifacts are text files like test and coverage reports. Some of these artifacts are binary artifacts like npm packages, jar files, and AMI machine images, which are built once and propagated along the CD pipelines for deployment in downstream environments.

CD pipelines generate a lot of artifacts. Before you know it, you're getting into terabytes, or tens of terabytes of generated artifacts. It's important to think through an artifact promotion strategy, which deals with where these artifacts are stored, how many are retained, and how you do clean-up.

The diagram above shows you an environment plan which factors in an artifact promotion strategy. From the plan you can see the environments that exist in the CD pipeline, with different colored arrows depicting different artifact promotion strategies. Early on in this pipeline, we generate artifacts that go to an artifact repository that has a more aggressive clean-up strategy, so we don't really retain too many of those environments. As you go further down this pipeline, artifacts are verified and more robust and you may want to keep these for a little longer, so you could keep them in a store with a different retention policy.

Dynamic environments

Environments are expensive and cumbersome to maintain. One way to make the process simpler and less expensive is to create environments on the fly. For example, when running functional tests, you can provision a functional test environment on demand, then clean it up right after your test is done.

To do this, you need use IaaC techniques to script all aspects of environment provisioning. Here are the benefits of doing this:

Parity with production If you create all environments dynamically based on automation, you will use exactly the same way to create testing and staging environments. The scale and nature of infrastructure may be different, but the footprint of the environment remains the same.

Prevent drift If you have manual processes and manual installations, you can't really ensure that two environments are the same. But using a script to create short-lived dynamic environments prevents drift.

Efficient use of hardware and infrastructure Only creating an environment when you need it and cleaning up dynamically optimizes utilization and saves infrastructure costs.

Technologies such as container schedulers are all the rage these days for deploying and running applications. The diagram below is an example of a Dock - a Docker based build workflow with Kubernetes.

The CI aspect of this pipeline, which is the build pipeline on the left, generates Docker images. The artifacts from this pipeline, which are Docker images, are stored in a Docker registry. Further downstream, deploy environments deploy directly to Kubernetes. You can leverage features of Kubernetes like its concept of labels to provision environments on the fly.

Summary

This is the part 4 of our CD of Microservices blog series. We have talked about environment strategy including artifact promotion and leveraging modern infrastructure for dynamic environments. In the next post, we will talk about the fourth consideration: configuration strategy.

Continuous Delivery of Microservices - Trunk Based Development and Feature Toggles

Sheroy Marker — Mon, 08 Apr 2019 16:51:48 +0000

This is the third post in the series - CD of Microservices. In the previous post, we talked about testing strategy for building CD pipelines on microservices architecture. In this post, we’ll get deep into CI practices.

Continuous integration

Continuous integration is a key practice in a successful continuous delivery strategy. Simply defined, it’s a practice that requires developers to integrate their code into a shared repository several times a day. Every checkin is verified by an automated build, allowing teams to detect problems early.

We’ll focus on two key practices, trunk based development and feature toggles. These two go a long way in implementing a simple and robust CI process.

Trunk Based Development

In trunk based development (TBD), developers collaborate on code in a single branch called “trunk”. The key benefit is to avoid drift in development branches and the resulting merge hell.

This is contrary to the practice of maintaining long-lived feature and release branches. In a branching model, though you may be running builds on individual branches, arguably you aren’t doing continuous integration. In trunk based development, you should never find your trunk in a state where your CD process is unable to deploy. All code should be checked into trunk, built and tested constantly, and the codebase deployable on demand: all of this make CD a reality.

TBD results in much simpler CD workflows: you don’t have to build multiple branches in parallel, map branches to environments and re-test when the same features get merged to trunk. Simplifying workflows is very important in the context of microservices, as the complexity is exasperated as the number of microservices grows.

Feature toggles is an important technique for TBD.

Feature Toggles

Feature toggles enable commits of a combination of work-in-progress and completed features. With these toggles, you can turn off the manifestation of incomplete features in production, until the features are dev complete and tested sufficiently in pre-production environments.

Here is a very simple example. The team are working on four features for the same application: search, menu, sign-in and in-app chat. The in-app chat feature is incomplete (but you still check into trunk in TBD) or you find issues with the in-app chat in pre-production testing. With feature toggles, you can simply turn in-app chat off, even at run time.

Feature toggles are usually stored in a specification or configuration file close to the codebase and used by automation in the CD pipeline to turn toggles on in specific environments. They are just conditions in your code base.You should separate setting these toggles from the actual release process so that you can control it in run time.

Some things you should consider when implementing feature toggles:

Feature toggles should be short lived

Feature toggles should be discarded once a feature has gone through the development lifecycle and is turned on in production. They are considered to be a code debt that needs to be cleaned up.

Use tooling to manage toggles’ lifecycles

Don’t underestimate the amount of effort required to manage these toggles. You can easily run into hundreds of them. Use tooling which provides visibility into the list of toggles, what’s turned on in which environment, and which features will get turned on in production in the next release.

Consider building your own utilities

There isn't a lot of tooling out there, so consider writing your own utilities to solve some of these problems. Again, getting this tooling in place before you go down a microservice strategy is a wise thing to do.

Once you have a mechanism for maintaining feature toggles, you could use the same mechanism to introduce other categories of toggles:

This diagram is borrowed from my colleague Pete Hodgson, who has written a lot about toggles. In this diagram, release toggles is defined as toggles that control access to unfinished code, which is same as feature toggles.

Ops toggles control the behavior of production code. Retail sites that have heavy seasonal traffic use Ops toggles to provide a degraded experience when they have peak loads. For example, when Apple is releasing a new iPhone and it's getting a crazy amount of traffic coming in just to buy the iPhone, they can turn off features like user recommendations during peak time to support the sales transactions.

Permissions toggles are used to turn on specific behavior for privileged users, such as admin features, or provide a guest user browsing experience.

Experimental toggles are used for multivariate testing. It is used to test how well a feature is received before you make it permanent. It's the same as A/B testing.

One thing to note is that each of these categories of toggles has a very different lifecycle and different way to be turned on and off. You need to plan accordingly. Release toggles generally are more short-lived and only live for the duration of a few releases. Once the feature is completely released, you get rid of the toggle and remove your technical debt. An ops toggle is used frequently for functionality in production, so it lives for much longer.

Summary

This is the part 3 of our CD of Microservices blog series. We have talked in depth about CI practices, in particular trunk based deployment and feature toggles. In the next post, we will talk about the third consideration: environment strategy.

Test Strategy for Microservices

Sheroy Marker — Wed, 06 Feb 2019 01:09:27 +0000

This is the second post in the series - Continuous Delivery for Microservices. In my previous post, I gave an overview of five considerations for building CD pipelines on a microservices architecture. In this post, we’ll get deeper into test strategy.

Test Strategy

A microservices architecture involves many moving parts with different guarantees and failure modes. Testing and verification of these systems are significantly more nuanced and complex than testing a traditional monolithic application. An effective test strategy needs to account for both testing individual services in isolation and the verification of overall system behavior. You can broadly break testing down into two categories: pre-production testing and monitoring and testing in production.

Pre-production testing of services

Here’s a simple example where you have build pipelines for multiple services and you're testing a service in isolation. In this case, the traditional test pyramid helps to maintain a balance between the different types of tests.

In a typical test pyramid, you have:

Unit tests

Tests that cover the smallest piece of testable functionality in your software.

Integration tests

Integration tests, in this context, deal with testing integrations and interface defects for components within your service; these are more granular tests.

Component tests

When you look at component tests for microservices, a component is a service that exposes certain functionalities. Therefore, component tests for microservice can just be acceptance tests for services and your tests need to validate whether the service provides the functionality that it promises to.

Contract tests

Another category of tests that's very applicable to microservices are contract tests. They test the contracts of APIs of your services to see if the API is valid or if the microservice honors its API. A cool variation of these contract tests is consumer driven contract tests. These tests are written by consumer services of an API; the consumers codify this contract in a suite of tests that get run on every change to the API. That way, if a change to the API breaks a contract that one of its consumers expect, this breaking change is caught early in the CD pipeline.

End-to-end tests

The test suites we discussed earlier are applicable to testing individual services. End-to-end tests, however, are more coarse-grained and try to test the functionality of an overall system. Depending on the deployment architecture you're going for, if you are deploying all of your services in a pre-production environment in an aggregate manner, you can run end-to-end tests there. Since end-to-end tests are usually brittle and take a long time to run, you’ll usually want to restrict the number of these tests to as few as possible. If you have microservices that are completely independent and don't get deployed to a pre-production test environment, then consider approaches that test in production.

Monitoring and testing in production

This traditional style of testing has its limitations. There are categories of errors that you can’t really simulate in test environments. Examples of these sorts of issues include issues caused by eventual consistency in a highly distributed system, and hardware and network failures causing parts of the system to fail. You have to supplement traditional testing techniques with techniques that allow you to profile and monitor systems in production effectively, and the ability to take remedial action in production when things do go wrong. In this post, I will focus on testing in production, and cover remediation strategy in a later part of this series.

There is a category of testing in production called fault-injection, which is introducing errors in a controlled manner in production to see if your system can hold up to those errors.

A variation of in-production testing are some specific deployment strategies that are popular in these environments:

Canary deployment

Canary deployment is where you take a new release and release it to a certain subsection of your production infrastructure, see how well that goes, and keep increasing the footprint of the new service until the time you completely roll it out. If you face issues, you can start rolling back the new version of your service.

Blue-Green deployment

Blue-green deployments are similar, where you have a new footprint of your new service, and then you do some testing and route some traffic through it. If everything is fine, you switch over all of your traffic to the new instance of services, otherwise, you keep the old footprint going.

Multivariate testing

Another interesting variation of this kind of testing is multivariate testing, where you're not really testing your new service against defects, instead, you are A/B testing new release features behind A/B testing toggles. The purpose of this type of testing is to see how well these features are received. You can decide roll it out to your entire set of users or make fixes where necessary.

Summary

This is part 2 of our Continuous Delivery for Microservices blog series. We have talked in depth about testing strategies for microservices, which include how to apply traditional testing pyramids to pre-production testing for microservices and also new techniques for production monitoring and testing. In my next post, we will talk about the second consideration: CI practices for microservices systems.

5 Considerations for Continuous Delivery of Microservices

Sheroy Marker — Mon, 04 Feb 2019 19:27:51 +0000

A microservices architecture builds software as suites of collaborating services. These architectures are generally accepted as a better way to build apps these days.

Continuous Delivery is an essential component of any software delivery practice. Regardless of the target deployment environment, you have to design a CD workflow to get software changes into production.

At ThoughtWorks, while partnering with clients building business-critical software, we overcame many challenges around building CD workflows for microservices. In this blog post, I will share the considerations we keep in mind in architecture design and application development.

Microservices and Continuous Delivery

According to Martin Fowler, microservices architectures are “a particular way of designing software applications as suites of independently deployable services.” These architectures are prevalent these days for building applications based on distributed systems concepts.

Jez Humble, in his pioneering book describes continuous delivery as "the ability to get changes of all types - including new features, configuration, bug fixes, and experiments - into production, safely and quickly in a sustainable way."

Regardless of the target deployment environment or your architecture choice, monolithic architecture in the past or microservices these days, it’s important to design a continuous delivery workflow to get your changes into production. A CD workflow is central to a DevOps process, and spans well across various functions in an organization including your development, QA and IT operations.

Four Challenges for CD on Microservices

Maintaining the integrity of complex distributed systems. Since you decompose a large monolithic system into smaller, more manageable microservices, the overall complexity of the system itself increases. You now have to deal with distributed systems concerns.
Safely and rapidly releasing features constantly. Managing frequent feature releases needs special consideration when your features could involve changes in one or many microservices.
Managing deployments of disparate technology stacks. Microservice environments often include disparate technology stacks for services. Managing a deployment process across these different stacks is challenging.
Process and tooling for deploying services independently and out of band. There are a lot of tools available to model CD workflows. It’s daunting to initially map out your CD workflow and pick tooling that best represents this workflow.

Five Considerations for CD on Microservices

There are five considerations I recommend keeping in mind when you design a CD workflow on microservices architectures. I will have an in-depth discussion for each of them in the following posts of this series. Here is just an overview:

1. Have an effective test strategy

Testing and verification of microservice systems is significantly more nuanced and complex than testing a traditional monolithic application. An effective test strategy needs to account for both testing individual services in isolation and the verification of overall system behavior.

For pre-production testing of services, especially in an isolated manner, traditional testing methodologies are still applicable and relevant. The test pyramid can still help you in maintaining a balance between the different types of tests. However, this style of testing has limited effectiveness when testing the aggregate of services. There are categories of errors that you can’t simulate in test environments, for example, issues caused by eventual consistency in a highly distributed system, hardware and network failures causing parts of the system to fail.

You have to supplement traditional testing techniques with techniques like synthetic user testing, lightweight user acceptance testing and fault injection testing.

2. Examine your CI practices

Continuous integration is a key practice in a successful continuous delivery strategy. Apart from the obvious considerations around build servers and build definitions, trunk based development and feature toggles are two key practices that go a long way in implementing a simple and robust CI process.

In trunk based development, developers collaborate on code in a single branch called “trunk.” The key benefit is to avoid drift in development branches and the resulting merge hell. This is contrary to the practice of maintaining long-lived feature and release branches. In a branching model, though you may be running builds on individual branches, arguably you aren’t doing continuous integration.

To do trunk based development, you need to have controls called feature toggles. Feature toggles enable multiple commits of a combination of WIP and completed features. With these toggles, you can turn off the manifestation of incomplete features in production, until the features are dev complete and testing sufficiently in pre-production environments. Feature toggles are usually stored in a specification or configuration file close to the codebase and used by automation in the CD pipeline to turn toggles on in specific environments.

Once you have a mechanism for maintaining feature toggles, you can use the same mechanism to introduce other categories of toggles like release toggles (to control access to unfinished code), Ops toggles (to control the behavior of production code), permissions toggles (to turn on specific behavior for privileged users), and experimental toggles (for multivariate testing - How well a feature is received before you make it permanent).

3. Plan your environments

An environment plan includes your sets of environments, the intended use of them, strategies to promote artifacts through these environments and toggle states on these environments.

First, think about what environments are needed and their intended use cases. Different groups in your organization will have different competing needs. When creating an environment, you should cater to all of these competing needs. Secondly, if possible, consider using cloud infrastructure to create environments dynamically. For example, use Kubernetes’ labels capability to create on the fly test environments for automated testing rather than have long lived environments. Thirdly, have an artifact promotion strategy. CD pipelines generate a lot of artifacts. You should you think about: how many artifacts to store, how many repositories you need, etc.

4. Manage configuration strategically

An application’s configuration includes everything that varies per deployment and should be stored separately from the code. How should you treat configuration when you have suites of microservices?

One technique we've seen to be useful is to manage deployment configuration centrally in repositories like Consul or Vault. Spreading deployment configurations across tools like Chef and the CD pipeline just makes it hard to understand and reason.

Another technique we use is to standardize processes for distributing configuration regardless of the technology stack of your services, and just let services handle the consuming of this configuration depending on the stack. For example, we generally use the 12-factor recommendations and avoid distributing configuration files.

And lastly, secrets like certificates need a governance process to ensure they are managed appropriately. This is usually a manual process but you need to think about it earlier and get it in place.

5. Prepare for things to go wrong

In microservices systems, multiple services get updated frequently, how do you respond when a deployment of a service introduces instability or bugs?

Roll forward, which means finding the root cause of a failure and applying the fix as soon as possible, is most times the best remediation response. A prerequisite for being able to do this is to ensure you have the capability to release from a hot fix branch straight to production. You may not want a fix to a production outage to go through the CD pipeline, depending on the time it takes for a change to make it through the pipeline.

Rollbacks are always tricky in production systems. In most cases if the change is granular and can be reasoned about, it’s easy to rollback. But if the deployment includes changes that aren’t easy to reason about, e.g., DB changes, especially ones that make schema changes, you need to deploy DB changes separately from code changes in consecutive deployments to ensure backwards compatibility of DB changes with earlier versions of code.

Summary

This is the part 1 of our Continuous Delivery of Microservices blog series. We have talked about four challenges and five considerations for building CD pipelines on microservices architecture. In the next blog, I will have an in-depth discussion on the first consideration: testing strategy for a system based on microservices architectures.