Forem: Kristin Baskett

Feature Flags and GitOps. 5 Use Cases to Help You ‘Git’r Done.

Kristin Baskett — Mon, 24 Feb 2020 19:10:14 +0000

Editor’s note: This article first appeared on Scotch.io.

The practice of GitOps is the next incarnation of continuous delivery. It allows developers to reach into the traditional purview of IT Operations – a land of many historical gates – and automatically update applications and the infrastructure they live on in production. In GitOps, the single source of truth for all change management and version control is the software configuration management (SCM). GitOps leaves behind traditional ITIL-type management and treats infrastructure and applications as versioned artifacts complete with the same granular audit trail (commit IDs, time stamps, etc.) that is captured during software development.

I do declare!

The idea of GitOps is that the desired state of the entire system is stored in version control via Git commits. If stripped to its bones, you can think of GitOps as a system for file versioning. A key tenet of GitOps is that the application and environment’s ideal state is described using declarative specifications via the use of a config file.

This means that the configuration is governed by a set of facts, not a list of instructions. Instead of giving you a recipe to build a thing, it gives you a rule to check that the result is correct. You can describe your entire CI/CD pipeline in the code repo this way. To make changes to the desired state, a developer issues a pull request which basically tells everyone about changes that you’ve published to the repo and instructs the repo to pull in the changes. By using Git you get version history, audit logs, rollback capabilities and the ability to see who changed what and when.

Feature flags + GitOps

When we consider GitOps, the use cases that come immediately to mind are container orchestration and cluster management – specifically with the declarative tool Kubernetes. Not many immediately think of feature flags. So why is this important that we consider feature flags? It’s important because the vision of GitOps is holistic control of the entire system. Feature flags are often perceived as living ‘outside the law’. We believe that feature flags – especially at scale – become an immensely powerful tool if they are managed with the same rigor, governance and importance as the rest of your code. If you want to use GitOps to manage feature flags, their desired state must be described using a config file. But that’s not all.

There is a right way to apply GitOps to feature flagging

Feature flags are a sticky wicket. They hold the power to make production changes, but they don’t undergo the same burden of proof for production readiness as the rest of your code. This needs to evolve if it’s going to become a staying part of the software delivery lifecycle.

If we want to manage feature flags with GitOps, the desired state (described by declarative specifications) must be saved to a config file. We use YAML so that it’s human readable and editable. When updates to the desired state need to be made you simply merge the config. This change is submitted via PR which establishes an audit trail and ensures that the correct folks are verifying the change – which is exactly how it happens when someone changes code in the application or updates infrastructure settings. We believe this is the right way to manage feature flags with GitOps. It’s also the truest to the vendor neutral aspirations of the practice.

To our knowledge, only CloudBees Rollout can support this. Some of our competitors also have a config file and their SDK knows how to read and change it. However, it’s not human editable. It’s also not automatically saved in an SCM like GitHub. They force users to circumvent the established process by which they manage code in order to manage feature flags. For example, if a feature rollback is required customers are forced to use a third-party dashboard, not Git where this should be taking place.

Some ‘Git’y use cases for feature flags with CloudBees Rollout

Configuration as code. This term is often used interchangeably with infrastructure as code (IaC), but it’s actually distinct. IaC is about the management and provisioning of the stack whereas CaC is about automating the migration of configurations between environments. This is all about making environment provisioning cookie cutter. No snow flakes allowed. We treat our feature flags the same way that configuration as code treats applications (we use CaC terminology here instead of IaC because feature flags are not part of the infrastructure, they are on the software application). When we’re talking GitOps this means we can document changes and versioning that happens in our feature flags in the same way the PR tracks changes and versioning for application changes in the SCM. Pushing changes to the main branch triggers an event to be consumed by the SDK. The system then understands how to update the YAML file configuration desired state for the feature flag.

CloudBees Rollout saves all feature flag and targeting data as local YAML files which are saved in a Git repository. Making changes to the local YAML files updates live CloudBees Rollout feature flagging data. We take advantage of Git’s characteristics as a decentralized version control system, allowing you full traceability and revision history even when working locally. If you edit your feature flags directly in GitHub and commit changes to the main branch, an event is triggered back to the dashboard and is also reflected in Rollout’s audit logs. If changes are done via the dashboard, the dashboard acts like a Git client and updates the YAML files on GitHub.

Once you are taking a configuration as code approach to your feature flags you can achieve these awesome use cases;

Governance and accountability: Because all changes are in Git, every commit produces an audit trail. You know who changed what in your feature flags and when. And because everything is managed by PR you can have team members approve your changes for extra accountability.
Progressive delivery, change and version control: Feature flags allow you to separate feature deployment from code releases. Eliminate long-lived branches by wrapping your features in a feature flag when they are committed to the main branch. They can remain in an ‘off’ state until they are code complete. Cutting down on branching in Git allows you do more progressive releases (increasing release velocity with bite size chunks). A GitOps approach to feature flagging ensures that every change is accounted for.
Cloning environments: With configuration as code you can clone feature configurations between environments (dev, QA, prod, feature X,Y,Z) by tracking feature toggle changes alongside your existing configuration.
Feature flag automation: When you have editable config file which describes the system desired state, it’s easy to run automation (for testing or deployment purposes) based on various desired states. You can automate deployment to a subset of the user base or a variety of segmentations using a GitOps approach to feature flags. When feature flags are set up as config files, it’s very easy to move the system to a new desired state. Other alternatives, such as the traditional CI process of using the rest API to change the feature flag, are more complex. Instead of waiting for authentication against the server, waiting for the network to report to the server and on… and on… using GitOps to manage feature flags is as simple as changing the config file in the Git repo to change the state.
Roll back feature changes with a Git command: There comes a time in every developers life when you need to roll back that commit. You can use feature flags to do that with a simple git revert command. Since CloudBees Rollout stores configuration code in Git, you can use branches to isolate changes to rollback in time and to work in parallel streams without affecting your production/Staging environment.

Even if adopting GitOps is still aspirational for your team, you can use CloudBees Rollout to manage your feature flags. API integration allows you to link to your favorite performance, analytics, monitoring and APM tools making it an easy fit regardless of how you manage the bridge between Dev and Ops.

Want to try it for yourself? Check out the 14-day free trial.

Feature Flags and Analytics: Putting your best features forward.

Kristin Baskett — Mon, 21 Oct 2019 14:01:15 +0000

Feature flags give you the flexibility to decouple releases from deployments. They give you the ability to toggle features on and off based on any number of criteria. But the question often remains, how do you know when it’s time to use them? How do you know when it’s safe to roll a feature out to the next target segment or all users? How do you know that a feature is misbehaving and it’s time to disable it? Application monitoring and analytics can help answer these questions and greatly increase the efficacy of your feature flag strategy. Let’s explore a few common feature flag use cases and how analytics can help developers make fast and informed decisions.

Early testing. One of the best uses of feature flags is to test features early in the development process. Effective testing against real user data in actual live end-user environments is critical. All the synthetic tools in the world can’t replace the real thing. When a feature is ready for limited availability or preview (or even Beta) feature flags can be used to make that feature available in production to a targeted subset of users. Often companies choose to first segment by internal employees and QA. The next subset would likely be those who have opted in as Beta users. Then you would advance to a targeted deployment of 5%, 10%, 50% or 100% for example to a group that may have criteria like; are new users from North America who use a specific feature in your application.

Late-breaking integrations. We’ve all been there. A feature makes it into a release by the skin of its teeth. Maybe you didn’t get the chance to test it as rigorously as you’d hoped before the release dropped. Some find the idea of testing in production exciting, but if that’s not you, never fear. You can wrap that late-breaking feature in a feature flag. If you discover buggy behavior, simple use the kill-switch and turn the feature off or pause it. This alleviates you from having to roll back a release for a feature you know might be shaky.

The analytics edge. For both these use cases you can use analytics to understand the user experience. Is the feature performing like you expect? When traffic starts hitting it, is the feature responsive? Does the feature improve or hurt my KPIs? Understanding these facets of user experience will help shape the way you iterate on features before deploying to the larger audience at GA. A good APM solution can give you this insight.

Canary deployment. In a canary deployment scenario a feature is released to a small targeted subset of users to see if it’s ready for prime time. The users on the canary deployment are gradually increased and tested along the way. If something goes awry, simply reroute the traffic back to the other version. Once all the users have been migrated, the canary deployment becomes the current software version. CloudBees Rollout supports this with gradual rollout.

The analytics edge. How do you know if something has gone awry? You want to know what’s going wrong and how systemic it is before pushing everyone to this release. Moreover, you really need to understand root cause. You can use APM and log analytics to help you understand metrics such as crash rates, error rates and latency. A good log analytics solution will be able to correlate metrics such as these to uncover root cause.

A/B testing. Similar to a canary deployment, A/B testing begins testing a feature with a subset of users and based on performance rolls it out to the rest of the population. However a Canary Release is more about feature stability and readiness. A/B testing is testing some hypotheses about user behavior. For instance, I may have a hypothesis that placing a product tour in a trial experience before a user connects their application will result in more conversions to a paying customer than if I were to place the same product tour after the user connects their application. This is an experience I can test with a feature flag. The experience that has a causal relationship with the outcome I desire is what gets rolled out en masse.

The analytics edge. Analytics for A/B testing can help you identify the causal relationship you are looking for. This is an especially exciting topic when you consider the application of A/B testing with behavior-based cohorting. One of our favorite tools, Amplitude, can create user segments by watching how users interact with your product rather than by title, industry etc. It allows us to test groups by how they behave, not who we think they are.

Analytics is an important part of a good feature flag management strategy. Whether you are integrating with APM solutions, log analytics or business intelligence software alongside your feature flags you are lightyears ahead of those still using feature flags as the “Oh sh*t! Turn it off! Turn it off!” button.