Forem: Armory

Building Trust in Your Deployment Tools

stephenatwell — Tue, 26 Apr 2022 22:22:06 +0000

The Cloud Native Computing Foundation found an extraordinary 300 percent increase in container use between 2016 and 2020. Kubernetes’ popularity is steadily rising, with 83 percent of respondents using it for production as of mid-2020.

Kubernetes’s out-of-the-box capabilities enable organizations to accelerate delivery cycles and rapidly scale their operation, contributing to its increasing adoption. However, organizations must support the container orchestration engine with secure and efficient release and deployment workflows to get the most out of it. The CNCF 2020 report highlights container release and deployment complexity among the top challenges the respondents faced.

DevOps engineers deploy to Kubernetes (K8s) by updating associated YAML files. While it may look easy for a single deployment, it quickly becomes arduous when performing multiple deployments across various environments. These include things like development, quality assurance (QA), user acceptance testing (UAT), staging, and production.

Although Kubernetes supports rolling updates, it’s challenging to do it manually since a Kubernetes application requires multiple YAML files. Manually building these files can be error-prone because of the white space syntax.

Also, handcrafting files requires substantial knowledge of Kubernetes components. These components can complicate the overall deployment process when you need to upgrade some to get the application working.

Auditing upgrades to track new and old deployments is also challenging to do manually, as is managing rollbacks for applications comprising several deployed services. Secrets management also becomes challenging when teams work on a distributed app across various repositories. Yet, containers must be secure to be trustworthy.

Many customers also desire more advanced deployment strategies than rolling updates in order to decrease the blast radius of a given change. Leveraging advanced strategies such as Canary Deployments and blue/green deployments ensures that if a new software version has an undetected issue, it affects as few users as possible.

Some of these challenges are unique to Kubernetes, but the other, broader issues apply to successful software delivery. Adopting continuous integration and continuous deployment (CI/CD) practices can address these challenges. Using tools to automate the entire deployment process reduces Kubernetes management effort so organizations can focus on core business needs.

Let’s explore how some of Armory’s automated tools help with CI/CD and address Kubernetes scaling and security concerns.

How Spinnaker and Armory Enterprise help with Kubernetes Scaling and Security

Self-serving enterprise development teams use continuous integration and continuous deployment (CI/CD) to deliver applications faster. These CI/CD practices enable business agility to give organizations a competitive advantage. CI/CD also leaves more time for innovation within your organization.

Rolling out applications to Kubernetes requires CI/CD solutions aimed at solving deployment challenges. Several open-source solutions help automate continuous delivery pipelines to Kubernetes, like Spinnaker, ArgoCD, and Flux CD. These solutions treat Kubernetes applications as first-class citizens, deploying the applications across various Kubernetes platforms, including Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS).

These CI/CD tools give organizations agility and control. They can implement various deployment strategies and perform rollbacks with all the required release governance.

Armory agent provides simplified and scalable management for all of your Kubernetes clusters when using Armory Enterprise or Open Source Spinnaker to manage your deployment pipeline.

GitOps

Armory Enterprise’s pipelines as code feature allows you to maintain your Spinnaker pipelines within git, supporting a GitOps strategy. GitOps leverages Git as the single source of truth and control mechanism for creating, updating, and deleting system architecture. First, Git provides permissions control, ensuring not just anyone can make changes. Second, because Git makes all the changes, organizations can define the infrastructure immutably to prevent other changes.

Organizations Face Kubernetes Scaling Issues

Open-source CI/CD solutions work well when organizations are early adopters, with small cluster sizes and a few deployed applications. However, they aren’t well-suited to large clusters.

For one, when a cluster contains many applications, the user can experience a delayed response to their action, and deploying changes becomes painfully slow. When the service is slow to respond to invocations by Kubernetes cluster components like kubelets, registries, API servers, and more, it causes timeouts and failed deployments.

Dashboards also often fail to capture deployment changes in near real-time and display stale information. The refresh slows as the application purges all the data.

Finally, these open-source solutions are not suited to scaling individual components to handle many clusters from the centralized instance. It’s challenging to understand the working details of the open-source solution.

One open-source CI/CD solution, Spinnaker, has an enterprise version that addresses these scaling challenges. Armory Enterprise, which is built on top of Spinnaker, works on clusters of any size, addressing scalability issues through improved redundancy. Its complementary features like infrastructure dashboards, drift monitoring, role-based access control (RBAC), audits, and more fulfill the development team’s needs. Armory also offers its Agent as a standalone offering for Open Source Spinnaker that increases its ability to scale to large-scale Kubernetes infrastructure.

Open-Source Tools Trigger Security Concerns

As open-source CI/CD pipelines automate and improve the software delivery workflow, they pose significant security risks. You need to trust that these solutions are not making your organization vulnerable to attack. Let’s examine some of these risks then see how an enterprise-level solution helps.

As enterprises prioritize software delivery speed, security and compliance are sometimes an afterthought. However, to trust their deployment tools, organizations must actively integrate security and compliance throughout their deployment pipeline. Their ability to adopt specific cloud-native tools depends on their current IT topology and organizational maturity.

Adopting a centralized CI/CD solution presents security concerns that require rigorous monitoring, command, control, and policy mechanisms. The CI/CD solution is a single source with access to all Kubernetes objects in all clusters. It often runs with full-access service accounts that can provision new components, stop existing services, read secrets, access external components including other clusters, and more. Additionally, it persists information to perform various analyses and generate dashboards.

Enterprise-level solutions like Armory Policy Engine can help centrally enforce policy on your CI/CD pipeline to improve security and make deployment more trustworthy. Additionally Armory Agent allows central management of Kubernetes accounts, helping you avoid proliferation of credentials that can access production.

Using Armory Agent for Trustworthy Deployment

Armory Agent is a departure from the centralized CI/CD solutions. It builds upon Spinnaker’s capabilities and makes the deployment process trustworthy at a massive scale — even when you have thousands of Kubernetes clusters.

The Agent is a lightweight, scalable service you deploy in a Kubernetes cluster. Each agent manages a single cluster so that you can scale the distributed approach for any number of clusters.

The embedded agent monitors all infrastructure and streams changes back to Spinnaker’s Clouddriver service. The distributed approach delivers several security benefits.

First, the Agent’s real-time sync captures the latest deployed changes. Also, since Spinnaker Clouddriver interacts with the agent, it decouples direct access to the Kubernetes API server, limiting a bad actor’s potential access.

Instead of centralizing account management, storing all Kubernetes accounts and credentials within Spinnaker, the Agent manages service account credentials in each cluster. This approach improves security by limiting teams to the clusters they manage.

Teams can manage permissions specifically for their clusters and quickly add new accounts without restarting Spinnaker. Finally, organizations can keep K8s clusters private with limited access since only the information Armory Enterprise requires leaves the cluster.

The Agent mitigates performance and latency issues associated with a centralized Spinnaker. It optimizes how infrastructure information is cached, making force cache refresh almost instantaneous. End-users benefit from optimal application performance, and DevOps teams benefit from optimal pipeline executions.

How Armory Project Borealis Improves Trust

Armory also offers an early access SaaS deployment offering called project borealis. It is a turnkey offering that incorporates many of our solutions from Armory Enterprise.

Project Borealis allows centralized management of Kubernetes clusters at scale. This allows you to control credential proliferation and manage all your clusters with a single toolchain. Project Borealis can also help you orchestrate your entire deployment lifecycle, across multiple environments, hosted across many different Kubernetes clusters. This helps ensure that your deployment process is consistently followed, decreasing risk and improving trust in your tools.

Project Borealis also provides advanced deployment strategies including Blue/Green Deployments and Canary Deployments. These strategies allow you to minimize the blast radius of your deployments, so that if the software contains an undetected issue the user impact, and any negative impact on trust, are minimized.

Project Borealis is currently available as an Early Access offering to our design partners. Spaces are still available, but filling up fast–so sign up today.

Summary

Kubernetes and cloud-native applications unlock business agility, but only when organizations support them with Kubernetes-specific CI/CD tools. Processes requiring manual intervention are error-prone and often cause bottlenecks. Organizations need automation to get the most out of their containerized services. They also need enterprise-level tools to trust that their deployments are secure.

Armory provides the secure, scalable, efficient, and reliable continuous delivery (CD) platform that you need. It builds on Spinnaker with policy-driven deployments that enable you to innovate with speed and safety.

Contact Armory today for a complimentary assessment of your software delivery practices and learn more about how your organization can benefit from safe, reliable deployments.

The post Building Trust in Your Deployment Tools appeared first on Armory.

How Progressive Delivery Drives Fast Feedback and Insights

stephenatwell — Mon, 25 Apr 2022 16:17:46 +0000

Software has grown exponentially across all industries in the last decade, both in adoption and complexity. This growth is due in part to digitization across sectors. It’s also in part because of the constant evolution of software development practices, which enable developers to build more exciting software better and faster.

Agile methodology has become the de facto standard for software development. Developers satisfy customers through early and continuous delivery of valuable software, then accommodate changing requirements throughout the development process.

Many companies have adopted some form of continuous delivery (CD) to reduce the feedback cycle, improving their software quality and development speed by steadily fixing bugs and adding features. Customers quickly get their requested changes instead of waiting months until the next software version.

We’ll explore how continuous delivery and progressive delivery enable a continuous feedback loop to improve software quality then discuss how Armory helps.

Continuous Delivery

Continuous delivery releases code changes automatically, either internally or externally. This makes a deployment repeatable, reliable, and predictable. When deployments are easy and reliable, it frees your developers to work on features instead of troubleshooting the deployment. They can then take the time to follow best practices instead of cutting corners on the deployment configuration.

So how can you use modern continuous delivery practices to improve software quality and delivery speed? By deploying often.

A modern application typically contains a few different components, each hosted on multiple machines behind a load balancer.

When your engineers release an application, they must deploy the code to not just one server, but all servers hosting the services. This method is both tedious and error-prone. Tools like Kubernetes help automate this process, enabling CD pipelines to push changes to the entire cluster using one simple command.

The ability to automatically push changes reduces your software development cycle’s feedback interval, increasing your software quality and delivery speed.

Progressive Delivery Drives Continuous Feedback

Progressive delivery builds on continuous delivery with two main additions: the use of feature flags to test code in production and progressive rollouts.

Many techniques have evolved to facilitate a rapid feedback cycle. Modern techniques like canary and blue-green deployments progressively release the application. These methods deliver the changes to only a subset of users, making it possible to safely test new features and gather data in a real production environment without a huge impact.

With progressive delivery, the development and release cycle typically looks like this:

Teams develop code, commit it, and prepare it for release.
They release the code to selected targets.
Companies gather relevant data through user feedback, telemetry, or statistics, and use the information to plan the next round of app improvements.
The cycle repeats.

One possible release strategy is to push the release to only internal stakeholders. Teams release the version to a broader public when internal stakeholders are satisfied with the changes. Customers won’t see the features stakeholders aren’t satisfied with in the release.

To optimize learning, progressive deployment can push new designs to a subset of real users and gather real usage data from a live production environment.

With the help of an observability tool, you can gather and aggregate all kinds of data relevant to your application, including logs, events, performance, and errors. Such tools collect data and also analyze the information to help developers resolve issues faster.

Now all the stakeholders have reliable and indisputable data to learn from. For example, your development team may have a solution to a performance problem, and they want to test it out by deploying to just 10 percent of the traffic through canary releases. As performance and usage data comes in, your team can roll out to more users and observe when performance degradation happens using real load. It’s usually challenging to simulate load using internal tools but possible with progressive delivery.

Iterative development is the engine of a rapid continuous feedback loop that cycles between development, feedback, improvement, and delivery.

Using a progressive delivery strategy can minimize downtime for each release. After-hours or midnight deployments are no longer necessary. Customers are happy, your release engineers are at peace, and your product analysts have the data they need to drive the next round of improvements.

Armory Enables CD and Progressive Delivery

Thanks to tools like Armory, developers can confidently release their changes to production and help stakeholders gather the information they need to drive the next round of iteration. Armory is an enterprise-grade version of Spinnaker, a continuous delivery tool built on best practices learned by delivering software at scale.

The software has many built-in features for you to implement progressive delivery with confidence:

Canary analysis is available out of the box. It’s easy to create a canary image before each deployment, collect some data, then either manually decide to push the release or make the decision automatically within a pipeline.
Once the application is released, you can undo the release through a pipeline or with a manual button click.
You can scale Kubernetes deployments. When the number of clusters grows, it’s typical to experience latency issues, and managing configuration and credentials also becomes a logistical nightmare. The Armory agent for Kubernetes addresses both concerns with a decentralized model and scales to thousands of clusters with ease.
Its policy engine ensures every rollout meets a set of codified standards. By setting appropriate policies, you can ensure every deployment is compliant with your organization’s standards, unifying DevOps and SecOps in the release process.

While you can implement progressive deployment with other tools like Kubernetes, Armory enables continuous delivery at scale.

For companies seeking a SaaS deployment solution, Armory also offers Project Borealis. Project Borealis delivers intelligent deployments-as-a-service so that your developers can focus on building great code, rather than deploying it. Project Borealis is currently available as an early access service to our design partners. Project Borealis is a drop in replacement for kubectl that can deploy using progressive deployments. If you’d like to learn more, sign up to become a design partner.

Conclusion

As software complexity has grown, the deployment process has become challenging. You can simplify this process with tools like Kubernetes to roll out changes to a cluster of servers at the click of a button, but that’s not all that continuous delivery is good for.

Today, progressive delivery powers many of the world’s best tech companies as the engine of a continuous learning cycle. Companies progressively roll out their product, gather accurate usage data in a real production environment, and use the information to help build their products.

Using Spinnaker as its core, Armory facilitates progressive delivery at scale, giving your developers the power to deploy with confidence. We invite you to check out what Armory can do for you.

The post How Progressive Delivery Drives Fast Feedback and Insights appeared first on Armory.

Dogfooding At Armory

Dan Peach — Mon, 18 Apr 2022 15:46:16 +0000

We recently onboarded the main server component of our new software delivery platform — codenamed Borealis — onto Borealis itself. I’m going to talk a bit about Borealis, the experience of dogfooding at Armory, and what we hope to learn from it.

Borealis and Deploy Engine

The application we onboarded is called Deploy Engine. The name is uninspired and is out of place among the sea of ostensibly Grecian software systems — Kubernetes, Argo, Tekton, etc. — but is descriptive of its role within Borealis: it orchestrates software deployment strategies.

Using the Borealis CLI or API, a user can provide a high-level description of their deployment: First, I’d like 10% of production traffic to go to my new software version. Next, I’d like the deployment to wait thirty minutes and then I’d like to run a canary analysis of application metrics. Next, I’d like to increase traffic to 50% — and so on. Deploy Engine, in concert with a few other subsystems, orchestrates this kind of deployment workflow.

Dogfooding at Armory

In my experience, the mental state of even the most empathetic, thoughtful, and experienced software engineer is divorced from that of a user.

I have two analogies:

I am a very poor actor but during high school was required to act in classes and in friends’ short films. The experience of using software for the first time as a user — rather than as a developer — is a little bit like the watching a recording of yourself as a bad actor. It can feel stilted or inauthentic; the transitions from one moment to the next can be painful.
Experiencing software for the first time as a user can mirror the experience of sharing a favorite video or song with a friend. What you once experienced as beautiful, catchy, moving, or hilarious can suddenly feel trite, bland, or tasteless as you hear the song through the ears of your friend. It can be jarring or unpleasant; it can also teach you about your friend and their values.

Dogfooding is hard, painful work, but it’s one of the best ways to experience your software as your users will. It’s not a replacement for user feedback — we have biases and beliefs that inform how we experience our software that may be different than those of our customers — but it’s an essential part of the development lifecycle.

What we’ve learned

I’ll share two anecdotes about our experience of dogfooding Deploy Engine.

When we first tried to onboard Deploy Engine onto the Borealis platform, we hit a strange bug that we’d never seen in testing or development. I soon discovered a bug that’s a little embarrassing, a little funny, and fortunately won’t take long to describe.

Our build uses Kustomize to generate manifests for five Kubernetes resources. When we tried using Borealis to deploy for the first time, only two of the five resources were deployed. We had unit tests that ensured that this case would work, so I was mystified. After some digging, I found the answer.

Our API accepts both JSON-encoded and YAML-encoded Kubernetes manifests. We check the first 512 bytes of a manifest: if those bytes look like JSON, we interpret as JSON; if not, we interpret the manifest as YAML. Unfortunately, in the case where we decided the manifest is encoded as YAML, we only used the first 512 bytes – we threw the rest away! Oops.

Our unit tests checked that we could deploy multiple manifests, but we used unrealistically small manifests — smaller than 512 bytes. This is a bug I’m glad a customer never had to experience, and is a bug we likely wouldn’t have caught without dogfooding.

My second anecdote is also brief. Borealis has a UI that lists all recent deployments; Deploy Engine determines how those deployments are sorted. Working with the product team, we defined the sort based on our experience and what we thought would make sense to customers.

Once we started using that UI as users, we realized the sort didn’t make sense at all — it bothered us, and we decided to change it. That very same day, one of our first users asked us to update the sort on the deployments page; their suggested change was the exact change we had just implemented. We were able to go back to the customer and tell them that their fix would be ready by the end of the day.

Today all the services used by Borealis are deployed using Borealis. Additional teams throughout the company have also started using it as they’ve seen it improve. Teams across Armory are giving us feedback. Some of it is tough to hear, but we’ll catch bugs and improve the product in ways that our customers will love. I hope to give you an update on Borealis, our progress, and our dogfooding journey in a few months.

If you would like to try out Borealis, we are still accepting design partners, and I’d love to hear your feedback on it as well.

Thanks!

Dan

The post Dogfooding At Armory appeared first on Armory.

Managing Compliance with Continuous Delivery

danielGz — Tue, 05 Apr 2022 15:52:49 +0000

This article explores how continuous delivery helps organizations maintain compliance and discusses how Armory helps with features such as rollbacks and progressive deployments to tighten control over your solution infrastructure. It shows how to improve code deployments, ensure compliance, and maintain a healthy and active innovation ecosystem between your developers and operations team.

Developers and DevOps teams are often stuck with highly manual, error-prone, and resource-intensive processes that do not always meet the requirements of various laws and regulations. Every industry has its own set of regulations and compliance rules. For example, the finance industry has extensive regulatory agencies and regulations, such as:

The Federal Risk and Authorization Management Program (FedRAMP), which applies to companies in the cloud
The Payment Card Industry Data Security Standard (PCI DSS), which regulates companies accepting online payments
Service Organization Control (SOC2), which helps establish trust in and transparency into an organization’s service delivery processes and controls

Additionally, some general regulations, such as the European Union’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), apply to organizations serving a regional or international audience. Organizations must vet every software iteration for compliance with these regulations before deploying fully.

Constantly changing regulations in industries like finance are especially challenging for organizations using a monolith system of application development and deployment. They can’t quickly meet changing compliance requirements and may face delays changing part of an application, which slows innovation. Monolith application delivery becomes challenging because you must run compliance tests on the entire tech stack for every change.

Many organizations turn away from monolithic applications and embrace continuous integration and continuous delivery (CI/CD) to meet regulatory and other challenges. Modern CD tools help technical teams quickly, and sometimes automatically, roll out changes to meet shifting regulatory requirements while maintaining audit logs.

Let’s explore how Armory improves your CD pipeline reliability and flexibility using progressive deployment strategies with one-click rollbacks to minimize risk. Additionally, we’ll discuss how automated policy enforcement ensures your team runs scans and follows configuration best practices.

Frequent Updates Using Continuous Delivery

Organizations are moving toward the microservices-oriented development and deployment model. This development model helps their engineering teams focus on individual challenges inside the entire solution and solve them quickly. Since each team maintains a single code base, it’s easier to regulate and ensure compliance. Also, in case of any problem, the team can easily roll back their deployment to a previously compliant state.

A typical enterprise application might comprise hundreds of small processes called microservices. Validating the compliance and regulation checks on hundreds of different applications is more manageable than one extensive application. This is because you can easily pin and regulate a non-compliant process during deployment checks. If a microservice isn’t compliant, the team rejects the deployment for that microservice only — not the entire stack. This rejection also alerts the developers responsible for the microservice’s maintenance to ensure compliance in their codebase.

Sometimes it’s not technically possible to debug and run the solution locally. For example, if your teams must provision and analyze the logs your app generates, it might not be feasible to run the entire cluster on a developer machine. However, provisioning a test or development environment for every team is expensive in licensing, hardware, and staffing. In contrast, with microservices, each team can run their project locally, ensure compliance, then push it for deployment. Your IT teams can run their deployment checks on each release to either approve or reject the ones needing improvements.

After you’ve passed your compliance checks, progressive rollouts work to minimize the impact of any undetected issues. Progressive rollouts direct a subset of traffic to the microservice’s new version while most traffic continues using the old version. This allows you to inspect the new version with a subset of the production workload, minimizing the impact of any undetected issues. When the versions have run in parallel without issue, you can scale up the new version and reroute the remaining traffic from the old version. If you discover an issue at any point in the deployment, one-click rollback enables you to reroute traffic back to the old version.

Continuous delivery ensures your IT and compliance teams are on top of every release to your infrastructure. In the following sections, you will learn how Armory can help them with production deployments, as well as with one-click rollbacks and progressive deployments. Although companies mainly use microservices to test and verify new functions in isolation, minimizing their footprint and decreasing software delivery risks, microservices also help your teams quickly apply changes as governing bodies introduce new regulations.

Flexibility and Reliability

Maintaining a continuous delivery pipeline can be challenging as you manage smooth rollouts and rollbacks. Continuous delivery requires you to test and verify your deployments for functionality and compliance. When your deployment doesn’t meet the production service-level agreement (SLA), you should be able to revert the changes to the previous working state.

Progressive deployments and one-click rollbacks help you control your DevOps pipeline. The progressive deployment makes sure your app rolls out to the targeted audience that you define. This function helps your IT teams gradually allow a group of customers to access your latest deployment, ensure everything is good to go, and then provide the newest deployment progressively to the rest of your audience.

The benefit of this staggered release is that you can control and quickly revert the application to a working state if your platform crashes or you find any bug. Armory’s one-click rollback, for example, helps you revert the recent changes to your infrastructure, such as Kubernetes.

With Armory, you can control when a rollback happens. This helps you prevent unnecessary rollbacks if something else in your production breaks. Armory is the enterprise-scale version of Netflix’s open-source software Spinnaker, and its features make your continuous deployments easier. Spinnaker supports major public cloud vendors, so you can easily connect Armory to any public cloud and start deploying your production stack.

When you find a compliance issue, you can revert your app to a compliant state using rollbacks. Rollbacks help you avoid inadvertently breaking the rules and incurring fines, losing a license, or other regulatory actions.

Controlling Continuous Delivery

The goal of continuous delivery is to prepare every code commit for deployment. Apart from verifying application functionality, you also need to run specific policy validations on your deployments to ensure security and compliance. The Armory Enterprise Policy Engine can run policy checks on your deployments to ensure security and compliance. Armory’s example policies can get you started.

You can also extend Policy Engine by writing custom policies using Open Policy Agent. These policies ensure compliance and make it easier for your compliance and IT teams to detect red flags in deployments. Your organization can write custom policies to ensure automated test suites and security scanners run before deploying your code to production. These policies include, but are not limited to, security bugs, hardcoded passwords or connection endpoints, logging user data, opening unnecessary ports, and more.

With this custom policy feature, you can control which images are allowed to deploy to production and which the tool should reject.

In addition to enforcing policies around your software delivery pipeline, Policy Engine can also enforce most policies that have traditionally been implemented with Kubernetes Gatekeeper. This decreases Kubernetes management overhead by enabling you to enforce your policies in a single place instead of independently in each Kubernetes cluster.

Some organizations don’t allow developers to access certain production logs. This lack of access slows down the deployment process, and when there is a production problem, the developer has no idea of what is going wrong. Armory with Spinnaker helps you maintain deployment logs while ensuring it doesn’t write sensitive information anywhere. You can then provide these logs to external or internal auditors.

Conclusion

In this article, we discussed the challenges apps face with compliance and regulations. Monolith app development and deployment are more likely to slow app deployments and releases by making it challenging to run compliance, quality, and regulation checks on deployments, slowing the entire DevOps pipeline, and making it challenging to roll back infrastructure changes. Continuous delivery helps overcome these challenges.

You learned how Armory’s DevOps and continuous delivery features, such as progressive deployments and rollbacks, help your IT team remain on top of every deployment. Controlling every deployment helps quickly catch any compliance issues before reaching your audience (and the attention of regulators).

The post Managing Compliance with Continuous Delivery appeared first on Armory.

How A Clear Strategy Guides Your Blitzscaling Practice

Thomas A. McGonagle — Wed, 30 Mar 2022 14:25:27 +0000

Colonel John Boyd was one of the best fighter pilots on the planet who risked his life every time he jumped into the cockpit. But thanks to his OODA loop strategy, high-velocity chases, death-defying rolls, and last-gasp pivots were no longer uncomfortable.
Almost 70 years after the Korean War ended, his aerial battle strategy is one of the most popular business decision-making frameworks in the world, used by many fast-growing companies to guide their blitzscaling tactics.
So, what does the military philosophy of a 1950s dogfighter have to do with the needs of fast-scaling startups today?
Let’s consider the four stages of the OODA loop:
Observe: Decision-makers must study their environment, identify changes at the micro or macro level, and determine whether a response is required.
Orient: With their observations at hand, decision-makers must position themselves to protect against threats or take advantage of opportunities.
Decide: Decision-makers must confirm if they are going to respond to the changing environment. While quick decision-making is encouraged, you should avoid acting out of any sense of obligation. Sometimes, the best course of action is to do nothing.
Act: Once decision-makers have made up their minds, they should act quickly.
Blitzscaling businesses can apply the tenets of the OODA loop to assess their current environment quickly, prepare for likely scenarios, make quick decisions and take action.
This strategy is an iterative process that encourages rapid decision-making and consistent optimization, as decision-makers will return to the observation stage to assess the results before repeating the process.

Final Thoughts
Whether it’s boxing, wrestling, or blitzscaling, real growth only happens when you consistently put yourself in uncomfortable situations. It’s only through practice that you can develop the situational awareness you need to adapt to changes.
When you're trying to move through the levels rapidly, things may seem chaotic without any clear or coherent plan. Whether 3 hours doing situps and punching things or it's grueling rounds at the gym, or endless dark mornings trudging through the snow with a bicycle heavy laden with newspapers, you can easily lose sight of your goals.

Taking a proactive approach to practice and applying Boyd's strategy of an iterative loop will give you some direction. You can continually observe your ever-evolving environment, aggressively solve problems, and make changes on the fly. In short, you must embrace the chaos, harness your most elite inner just like a fighter pilot.
I love to talk about fighting and blitzscaling. If anyone is interested, please do reach out. My work email is thomas@armory.io, my calendly is https://calendly.com/thomas-mcgonagle and my cellphone is 339-203-3816. You can call anytime.

9 Tactics to Guide Your Blitzscaling Practice

Thomas A. McGonagle — Wed, 30 Mar 2022 14:20:42 +0000

Ari Libarikian is a senior partner of Leap, McKinsey's business-building capability.
He explains that “the companies that have stayed at the top for decades are always regenerating. This is really a requirement to being successful today.”
Serial business builders, like Amazon, Google and Facebook, are always regenerating. They adapt to changes in their environment and constantly launch new products, services and business ventures. In other words, they have mastered the art of blitzscaling.
For that to be possible for your business, you have to forget almost everything you learned in business school. Here are nine counter-intuitive principles of blitzscaling to guide your efforts in practice:

**Embrace the Chaos
**Making speed paramount at the expense of efficiency, order and stability have to go out the window means things will get a little haphazard. You must accept that your business is in for a bumpy ride in the quest for product-market fit. Turbulence can be fun and is fun!
Hire Ms. Right Now, Not Ms. Right
Some people might be a great fit for the latter phases of blitzscaling but less suited to the earlier startup phases. Aim to recruit the right people at the right time. Similarly, be prepared to let people go as you grow.
**Practice “Bad” Management
**If you're truly embracing the chaos, the structure of your organization will bend and mold multiple times each year. Worry less about keeping things organized, so you prepare to handle unexpected challenges.
Launch a Product That Embarrasses You
A perfectionist won't succeed in this game. You need to jump in the ring and start making mistakes. You might get embarrassed, but the sooner you get your MVP into customers' hands, the quicker you can get feedback that helps you iterate and improve. I was an awful wrestler, until one day I wasn’t. Be awful until you aren’t.
**Let Fires Burn
**The second of Kravernetes Tenants is to “be vigilant in identifying and addressing challenges.” When you’re trying to grow quickly, you won’t be able to handle every problem. Focus on the key strategic areas that will fuel your growth or the major issues that could potentially ruin you. Forget everything else.
**Do Things That Don’t Scale
**When the Airbnb founders went banging on doors doorknocking to take quality photos of their listings, they knew it wasn't a scalable solution. But sometimes, before you can scale, you have to do the things that matter by any means possible.
**Ignore Your Customers
**Getting feedback from customers will help you improve your product with every iteration. But you must figure out how to scale your customer service to appeal to a broader audience, which will mean letting some issues go in the early days.
**Raise Too Much Money
**A defining aspect of blitzscaling is the inefficient use of capital. Having a large war chest is a defensive tactic to protect scaleups from crashing and burning—but it can also work offensively. Having deep pockets will enable you to take advantage of new opportunities that arise as you scale at speed.
**Evolve Your Culture
**Startup founders will set the tone for company culture, but it’s by no means set in stone. As your company grows and adds more people, the culture will naturally change.

Reducing DevOps Burnout with Next-Gen Continuous Delivery: Progressive Delivery

mashby2022 — Mon, 28 Mar 2022 15:03:56 +0000

The development experience can be interesting and enjoyable, but it can also be daunting. From writing code to committing new features and fixes to getting them working in production, the work we put into developing a software application is brain-intensive and challenging.

Software engineers face many challenges. They regularly deal with changing requirements and fresh demands from clients. Engineers also have to learn and understand new frameworks that enhance the development experience, updates to existing frameworks, new language features, and many other innovations.

DevOps teams have to find ways to continue delivering top-notch services that keep the client happy while not burning out from work stress. Knowing that the client’s happiness ultimately lies in the hands of the software team, technology companies must do their best to reduce drudgery in the development cycle. Updating and modernizing processes and tools, and emphasizing automation, helps reduce this tedium.

A key way to bridge the gap between motivated, healthy developers and happy customers is to pay attention to the “beautiful chain” connecting them: the continuous delivery (CD) pipeline.

Let’s explore some contributions to DevOps burnout and discuss how progressive delivery, the next generation of continuous delivery, helps reduce this burnout for happier workers (and happier customers).

Contributors to DevOps Burnout

Customers often have strong demands for new features, changes, and fixes. And they have high expectations for the speed said updates are delivered and stability with which they perform. Satisfying that demand falls on the shoulders of developers. There are four delivery metrics that reflect a development team’s ability to deliver against these demands.

First, there’s deployment frequency. How often does your software team release new changes into production – daily, weekly, or monthly? Second is the lead time for changes. How long does it take for your team to deploy a commit into production? Third, your change failure rate. What is the ratio of failed deployments to the total number of deployments? Finally, customers are concerned with your time to restore services. How long does it take to restore services when a deployment fails?

SlashData research shows that only about 10 percent of developers release multiple deployments per day. Nearly two-thirds of developers take at least a week for their committed code to run in production.

Some development lifecycle lapses result from poor practices and tools. When developers manually do processes that they should instead automate, it adds to their mental and physical stress. Your development team can be composed of great software engineers, but if your company fails to give them the right tools they need to do their work, productivity and efficiency drop drastically. Of course, this in turn drops the quality of service delivery.

Additionally, software teams relying on manual processes live in constant fear of being paged at midnight. Chances are, a priority customer is using the software for crucial work when the application crashes. And this can be simply because a bug found its way into production undetected.

As more people worked from home, businesses moved their processes online. This digital shift caused increased workload and anxiety for development teams. Statistics show that about 83 percent of developers and DevOps professionals are experiencing burnout.

As researchers dug into this burnout’s root cause, they traced it to cycle time. Developers are under intense pressure to reduce their cycle times while maintaining or, better still, improving the application’s quality. Trying to meet these demands, especially with flawed tools and processes is one of the biggest causes of DevOps burnout.

Developers can mitigate this burnout by embracing better practices like effective communication, taking regular breaks, sticking to dedicated and properly regulated working hours, and using the correct tools. Of course, you need your organization’s support to achieve this.

Software delivery requires continuous improvement. Organizations and individuals who fail to improve will fall further and further behind. If companies really want to ensure that their clients are always happy without sacrificing the health and well-being of their developers, they need to pay close attention to DevOps practices.

How Progressive Delivery Helps Reduce DevOps Burnout

Progressive delivery is a technique for managing new application releases. Organizations can deploy new functionality to a subset of users before gathering feedback and delivering these new functions more widely. Progressive delivery refers to deployment best practices.

Much of the pain development teams experience while deploying solutions arises from faulty DevOps processes and tools. When processes that should be automated are manual, it increases the workload for development and DevOps professionals. This heavy workload causes burnout and other issues.

Manual DevOps practices introduce human errors, especially in large deployments. These errors include misinterpreting a line of text, missing an error code, or making changes directly on the server. These “minor” mistakes can cause grave issues, like a system crash, deployment to the wrong environment, or changes in one section of an application causing breaks in another part.

Developers get trapped in a loop of endless fixes and deployment errors. Development and DevOps professionals experience burnout at the beginning of the loop due to manual processes that should be automated and at the end of the loop. This is because they still have to bear the brunt of all the other issues that crop up. They must find ways to fix these issues as soon as possible while simultaneously struggling to add new features.

Progressive delivery helps resolve these issues and reduce development team burnout in many ways.

First, progressive delivery replaces manual processes with DevOps automation. Technologies that perform tasks with minimal human intervention facilitate DevOps automation. Automation also enhances the feedback loop between development and operations teams, making deploying application updates into production iteratively easier and faster.

Ops personnel no longer need to babysit deployments to ensure they go smoothly. This automation leads to increased business value and overall improvement in service delivery quality since it minimizes human errors and development teams are free to focus on their core task of developing application functions. The fully automated CD pipeline does all the low-level work. When focusing on the work they love, developers are much less prone to burnout, and everyone is happier — including clients.

Deployment Techniques

Multi-environment orchestration enables you to perform more reliable pre-deployment testing before pushing changes into production. This includes running integration test and security scanners in staging before the new version ever reaches production.

Deployment strategies, such as blue-green and canary, enable you to test your deployed production software before exposing it to full production traffic. With blue-green deployment (also called red-black deployment), DevOps runs two production environments side by side: one as a test server and the other as a live server. These functions are interchangeable.

The development team tests new updates on the test server while the client workload remains on the live server. As soon as the new deployment is certified production-ready, DevOps switches the workload from the live server over to the test server, making it the new live server. The previous live server becomes the test server for the next set of updates.

Considering that both servers are production environments, it’s straightforward to detect any inconsistencies or bugs and resolve them before going live.

The canary deployment technique, in contrast, progressively increases how much production traffic is reaching the new version of the software. This is often implemented by deploying updates to smaller subsets of production servers before cascading them to the rest of the servers. After each increase in traffic the logs and monitoring can be checked to ensure that production traffic is not causing unexpectedly high error rates, or unhealthy behavior on the new version. With this technique, DevOps can deploy, for instance, to 5 percent, 10 percent, 25 percent, 50 percent, then 100 percent of servers. This method ensures that if a bug or system glitch makes it into production, it affects only a few users.

The blue-green and canary deployment techniques help reduce downtime to the barest minimum, ensuring that the live application is in the best possible state at every point in time.

Progressive Delivery’s Benefits

Progressive delivery tools and techniques strive to eliminate the need to roll back changes by facilitating high-quality deployments. However, there may be cases where you need to revert to a previous build. This isn’t necessarily because of a bug or glitch — you might want to go back in time for any other reason. In such a situation, most progressive delivery tools provide features to roll back changes quite quickly.

Following progressive delivery techniques and deployment best practices make the whole deployment process seamless, reduces human errors, and minimizes the need for unnecessary rollbacks. These techniques help ensure development teams stay happy and motivated rather than burnt-out. Plus it keeps the customers pleased as well. It’s a win-win!

Embracing Progressive Delivery Solutions

Armory, a premier member of the Continuous Delivery Foundation, offers an enterprise-grade version of Spinnaker, the world’s leading Continuous Delivery solution. This CD tool empowers developers to deploy instantly and to the cloud, ensures compliance, and promotes better security. Armory also helps engrain the culture of progressive delivery into the software development lifecycle, minimizing DevOps burnout.

Armory ships with blue-green and canary deployment features. These deployment styles give development and DevOps teams the confidence to deploy without surprises from sudden failures.

In the unlikely event that a failure does occur, Armory also automates rollbacks with one click. This automation ensures that the efficiency of the feedback loop between customers and the development team remains top-level. DevOps burnout is reduced to the barest minimum, leaving the developers and operations professionals happy and motivated. Clients are also delighted as they meet their business objectives — and satisfied clients generally increase your organization’s revenue.

In addition to Armory Enterprise, Armory recently announced a new standalone offering that provides continuous-deployments-as-a-service. This offering, Project Borealis, is focused on meeting companies where they are at in their CD journey and simplifying the most common progressive delivery practices. The average time for a design partner to start leveraging canary or blue/green deployments is under an hour, and that includes design partners who have never done progressive delivery before. If you are not ready for or don’t need Spinnaker, but want a simple progressive delivery capability that you can easily trigger from your existing toolchain, Project Borealis is still accepting new design partners. Spaces are limited, so sign up today.

Summary

Technology is changing rapidly, and if you’re not improving, you’re getting worse. You should review your stack to ensure you employ the right tools and techniques. Manual DevOps processes lead to burnout and ultimately reduce the quality of service delivery to your clients, increasing the risk of losing them to competitors.

One of the critical causes of DevOps burnout is manual processes. DevOps employing progressive delivery experience a clear difference.

Armory helps you improve your continuous delivery process efficiency and your overall DevOps ecosystem. Contact Armory today for a complimentary assessment of your software delivery practices and learn more about how your organization can benefit from safe, reliable deployments.

The post Reducing DevOps Burnout with Next-Gen Continuous Delivery: Progressive Delivery appeared first on Armory.

Elite Stability: How Armory Delivers Customer Satisfaction

Ben Mappen — Wed, 23 Mar 2022 16:10:26 +0000

On October 4, 2021, the world came to a halt. Over 3.5 billion people were impacted, as a service outage caused one of the most popular websites in the world to vanish from the internet.

Facebook was down. Not only that, the outage crippled Meta Platform Inc’s entire family of apps, including WhatsApp, Messenger, Instagram, and Oculus. In an instant, billions of people lost access to their main global communication channels, and millions of businesses lost their ability to sell on social platforms.

As Facebook employees scrambled to find the issue, companies all over the planet began counting their losses—which grew by the minute.

Roughly seven hours later, Meta restored full service. A fault in the underlying internet infrastructure interrupted the communications between the data centers.

The cost?

Facebook lost $60 million in revenue
Meta’s market cap plummeted by $47.3 billion
Mark Zuckerberg’s net worth dropped by almost $7 billion

That’s right, the main man himself lost nearly a billion dollars an hour from his back pocket. Ouch. And that begs the question: how much would an hour of downtime cost your business?

Let’s look at how the economic impact of service outages can hit your customers and how you can safeguard against disaster and drive customer satisfaction.

How much does a service outage cost your customers?

Awkwardly, Facebook chiefs had to use Twitter to apologize to the world publicly and explain they were working hard to get things back up and running.

We know Zuckerberg and his company were severely impacted—but what about the customers? How much do customers lose when a software company suffers a major service outage?

According to research from July 2021, an astounding 98% of organizations claim that a single hour of downtime costs over $100,000. Think about that for a moment. How would your business cope if a service outage knocked your lights out for a full day? Would you even survive?

CNBC reports that over 10 million brands and businesses advertise on Facebook’s platforms. When these apps went down, they all lost the mainline to their customer base. Some companies saw sales nosedive during the outage, reporting losses of 30% to 70% compared to the previous week.

Creators and social media influencers felt the pinch, too. While some lost just a few hundred dollars, others were left to rue the loss of over $5,000 they could have made through sales, product launches, affiliate links, and sponsored posts.

If a blackout like this can hit one of the biggest software companies on the planet, it can happen to anyone. When a service outage hit Amazon Web Services in December 2021, it disrupted a host of popular services, including companies in retail, airlines, stocks, and crypto.

Complaints rolled in from users of Netflix, Roku, Disney+, Tinder, Venmo, Robinhood, League of Legends, Kindle, Instacart, and the McDonalds app. MarketWatch reported that people couldn’t book flights with Delta Airlines and Southwest Airlines.

It was bad enough not to be able to spend frivolously on Amazon. But people were having issues shopping on booking flights, ordering Big Macs, binging their favorite shows, or even swiping right on the guy or girl of their dreams.

As you can imagine, service outages like this can make a lot of people pretty damn unhappy. That’s not good news for any software company.

The undeniable importance of customer satisfaction

Customer satisfaction is a measurement of how happy customers are with the products, services, and capabilities of a company. It’s not a fluffy, meaningful metric—it’s a key performance indicator (KPI) companies can’t ignore in the digital age.

Customer relationships have changed substantially in the digital age. The concept of stickiness due to switching cost hasn’t died . . . but it’s severely less pervasive than in the past. Customers are spoiled with choice like never before. If you can’t make them happy, someone else will. Customer satisfaction tells us how good a company is at meeting the expectations of its customers.

Software companies that fail to remediate bugs and service disruptions will inevitably face complaints rapidly. The longer a poor user experience impacts your app, the more damage it can do to customer satisfaction. If the problem persists, users may leave and might never return. After all, whatever niche you’re in, there is no shortage of alternatives.

PWCreports that 32% of customers will quit doing business with a brand they love if they have just one bad experience. Think about that prospect for a second—you could lose almost a third of your entire customer base after one high-profile fiasco.

Stock investing app Robinhood found itself in deep waters after some questionable service restrictions left users unable to access their accounts. Disgruntled customers couldn’t reach customer support, so they stormed the fintech’s HQ in California. From vandalism to people throwing dog feces, the debacle tanked the company’s reputation. The Financial Industry Regulatory Authority (FINRA) slapped Robinhood with a record-breaking $70 million fine before the startup went on to make one of the worst stock debuts ever.

On the contrary, when you excel at meeting expectations, there is a lot to gain in the long term. Happy customers will engage your business more often, buy more, and become loyal ambassadors for your brand and products over time.

Sounds simple, right? Give them what they want, when they want it. The big question is: how?

The answer: Elite stability.

How to prevent service outages and maximize customer satisfaction

The bar is set high for software companies, as people are accustomed to lightning-fast apps driven by automation and real-time data processing.

And so, DevOps is crucial to customer satisfaction. A well-oiled machine for continuous delivery is the backbone that provides elite stability for software apps in 2022. And that’s where Armory can help.

Armory was built to enhance Spinnaker with the features that enterprise development organizations need. It provides a collaborative delivery environment making software delivery continuous, collaborative, scalable, and safe.

Here’s how Armory provides that elite stability:

High resilience. When you make small, continuous software changes while customers are actively using your app, a lot of things can go wrong. Armory acts fast and effectively, taking less than one hour to restore service and maintaining a change failure rate below 15%.
High transparency. Security is great to have—but giving visibility to users takes the trust to another level. Armory operates with a “Single pane of glass” to provide users an end-to-end view across an application, connecting clouds to pipelines to artifacts, all the way back to the code.
Maximum productivity. We empower developers to increase productivity with standardized templates they can use to create repeatable and scalable pipelines. This approach helps accelerate builds and reduce investigation time in the wake of an outage.
Continuous improvement. The Armory policy engine helps companies create and enforce policies to standardize pipelines and reduce the number of failed deployments.

Final word

A service outage is no joke. Amazon lost about $100 million per hour during downtime on Prime Day. But even after you restore your service after an outage, the damage can run much deeper. You might lose users and the trust of valued customers who use your service to run their business.

When people rely on your software to run their business or communicate with others, they expect constant uptime. Nobody has time for service outages—even a few minutes can be devastating.

If you want to keep your customers happy and maintain a competitive edge in your market, you need to establish a reliable system. Continuous delivery with elite stability can make-or-break your company’s reputation. Keep your service up to keep up with the customer’s expectations, or you might lose them forever.

Ready to put a smile on more customers’ faces? Get in touch with Armory to see how we can help you create more stable, reliable applications.

The post Elite Stability: How Armory Delivers Customer Satisfaction appeared first on Armory.

Blitzscale Tactics: Relentless Practice Over Perfect Planning

Thomas A. McGonagle — Wed, 16 Mar 2022 12:14:03 +0000

Anything worth doing in life is hard. It doesn’t matter whether it’s a paper route in the Boston winter, a boxing match fight or blitzscaling your business—you have to get outside your comfort zone and to grow.
Blitzscaling, which prioritizes speed over efficiency when growing your company, will push your team into past their comfort zone unfamiliar territory. As you prioritize speed over efficiency in uncertain environments, you will make mistakes. This relentless advance moves at an uncomfortable pace for many people. But the only way to learn from your mistakes and grow is through constant practice, practice, practice. And once you are done practicing. Practice some more.
Why Practice Matters in Blitzscaling
Anyone can talk the talk but it's the walking rightously that matters. If you want to sprint run, you need to do more than talk. You need to play the game. The more you play, the better you get, and the better you get, the more you play. Your team must take consistent action to test new ideas and hone new skills.
Kravernetes Tenents advocate using simple and repeatable techniques and invoking precision when executing these tasks. The truth is that precision will only come when you repeat a technique 10,000 countless times.
The best fighters boxers in the world understand this concept. Every day they trainin training, they practice the same techniques over and over relentlessly. The same foot movements; the same punches; the same combinations. For new fighters newcomers, it feels like overkill alien at first, but repetition enforces muscle memory breeds conviction. Through continuous drilling, everything begins to happen automatically. And then, in the match fight, a the boxer can go into war God mode and rely entirely on rely on muscle memory. In wrestling for example, I found the threshold somewhere around practicing a move 1,000 times before being able to do it in a match.
It sounds ridiculous but I shadow box 3 hours a day and have for most of my adult life. I am often surprised by just how many people don’t know what wins fights….Particularly marines never know this. Its not part of their training which is somewhere south of 20 hours of hand to hand training for the average leatherhead. It is simple. Footwork wins fights every time. It doesn’t matter what the sport. Footwork trumps everything. A fight completely changes every time you slide to the right. The angles change. The way you win a fight is you control the angles and your position against your opponent. WYou can literally kill someone with just a simple left jab. Punch him in the nose, take a step back. Do it over and over again, and I guarantee they will eventually stop coming at you. Don’t get me wrong, I have thrown my share of haymakers, or even suicide punches, but they are easy to block and leave you exposed. Don’t get me wrong they have a role and can definitely make a point. But what wins fights is footwork.

Second Blog

In the startup world, the companies that consider all the angles and train hard to become more agile will be the ones that can move fast and are ready to respond to changes. Being slow to react or making the wrong decisions under pressure might not result in you getting brutally separated from your consciousness—but it could mean a knockout blow for your business. The hot blood rush of a concussion can be difficult to deal with, especially when everything turns grey…My advice for startups and individuals is simply don’t get punched. Stay out of reach. Move. Slide. And constantly change the angles.

9 Tactics to Guide Your Blitzscaling Practice
In the startup world, the companies that consider all the angles and train hard to become deadly will be the ones that can move fast and are ready to respond to changes. Being slow to react or making the wrong decisions under pressure might not result in you getting brutally separated from your consciousness—but it could mean a knockout blow for your business. The hot painful blood rush of a concussion can be difficult to deal with, especially when everything turns grey…My advice for startups and individuals is simply don’t get punched. Stay out of reach. Move. Slide. And constantly change the angles of the attack.
Ari Libarikian is a senior partner of Leap, McKinsey's business-building capability.
He explains that “the companies that have stayed at the top for decades are always regenerating. This is really a requirement to being successful today.”
Serial business builders, like Amazon, Google and Facebook, are always regenerating. They adapt to changes in their environment and constantly launch new products, services and business ventures. In other words, they have mastered the art of blitzscaling.
For that to be possible for your business, you have to forget almost everything you learned in business school. Here are nine counter-intuitive principles of blitzscaling to guide your efforts in practice:
Embrace the Chaos
Excuse my French, but I fucking love chaos. I was designed for War. Business War requires Mmaking speed paramount at the expense of efficiency, order and stability have to go out the window means things will get a little haphazard. You must accept that your business is in for a bumpy ride in the quest for product-market fit. Turbulence can be fun and is fun!
Hire Ms. Right Now, Not Ms. Right
Some people might be a great fit for the latter phases of blitzscaling but less suited to the earlier startup phases. Aim to recruit the right people at the right time. Similarly, be prepared to let people go as you grow.
Practice “Bad” Management
If you're truly embracing the chaos, the structure of your organization will bend and mold multiple times each year. Worry less about keeping things organized, so you prepare to handle unexpected challenges.
Launch a Product That Embarrasses You
A perfectionist won't succeed in this game. You need to jump in the ring and start making mistakes. You might get embarrassed, but the sooner you get your MVP into customers' hands, the quicker you can get feedback that helps you iterate and improve. I was an awful wrestler, until one day I wasn’t. Be awful until you aren’t.
Let Fires Burn
The second of Kravernetes Tenants is to “be vigilant in identifying and addressing challenges.” When you’re trying to grow quickly, you won’t be able to handle every problem. Focus on the key strategic areas that will fuel your growth or the major issues that could potentially ruin you. Forget everything else.
Do Things That Don’t Scale
When the Airbnb founders went banging on doors doorknocking to take quality photos of their listings, they knew it wasn't a scalable solution. But sometimes, before you can scale, you have to do the things that matter by any means possible. Courage and bravery are important. The Gods will reward you. .
Ignore Your Customers
Getting feedback from customers will help you improve your product with every iteration. But you must figure out how to scale your customer service to appeal to a broader audience, which will mean letting some issues go in the early days.
Raise Too Much Money
A defining aspect of blitzscaling is the inefficient use of capital. Having a large war chest is a defensive tactic to protect scaleups from crashing and burning—but it can also work offensively. Having deep pockets will enable you to take advantage of new opportunities that arise as you scale at speed.
Evolve Your Culture
Startup founders will set the tone for company culture, but it’s by no means set in stone. As your company grows and adds more people, the culture will naturally change.

Reducing Downtime Risk with Blue-Green Deployments

Dan Peach — Tue, 15 Mar 2022 15:35:39 +0000

Many of the world’s most common and trusted applications- for both personal or business use – are now delivered as SaaS applications, whether on mobile, desktop, or other devices. End users assume these SaaS applications – whether paid for via subscription or invoked free on demand – to be available 24/7/365. Users trust application builders to provide a dependable and uninterrupted service experience. When software isn’t available, customers get frustrated and businesses’ reputations suffer.

Leading content platform and application software companies are increasingly using progressive software deployment strategies such as ‘blue-green’ to reduce the risk of downtime, enhance customer experience, and build brand loyalty. However, in many companies, there still exists the misconception that initiatives to improve quality and security translate into increased investment. Therefore, a business case for making the technology investments required to significantly reduce the mean time between service failures to avoid downtime will likely be required.

The Costs of Downtime

According to a 2019 report from Statista, the average cost of downtime can range from $100,000 to $5,000,000 per hour.

Content customers and software users are working all hours, locally and around the world. Therefore, there is no ‘good’ time to have an outage. They expect and/or require the applications to be available all the time.

A recent Accenture study found poor service quality is the top reason for customer churn. In addition to compromising customer experience, jeopardizing renewals, expansions and upsells, excessive downtime can trigger direct costs if service-level agreements (SLAs) are in place for operational performance. In addition, if a service is delivered to regulated industries like banking, insurance, and stock trading, outages may violate compliance requirements. Said violations can result in fines, penalties, or increased scrutiny from auditors and regulators. Resultant scrutiny is very expensive in terms of time, consuming resources that could be working on revenue generating activities instead.

There is also a sizable intangible cost to organizations that face continuous downtown incidents. Service breakdowns impact employee productivity, morale, and job satisfaction. This often results in higher employee turnover.

Implementing practices to achieve service uptime performance consistent with customer expectations is critical to sustained success. Blue Green deployments are a technique increasingly viewed as an operational best-practice for deploying applications reliably into production and ensuring minimal downtime.

Blue-Green Deployment

During a blue-green deployment, the system shifts traffic routing from the currently-deployed version to a new application version. This practice requires two concurrently-running application instances:

A current, or “blue,” version of the application
A new, or “green,” version of the application

Containerization and cloud-native development augment the blue-green deployment model: Kubernetes provides a cost-effective, flexible, and automated process of generating similar application instances, and label-based traffic routing provides the necessary foundation to switch between different application versions quickly.

I work on Armory’s new SaaS-based deployment offering, Project Borealis. We just added blue-green support. In Borealis we deploy the new version, then expose it to a preview or “dark” URL that the user can leverage for any needed validation. Once validated, they can issue an approval, and the new version will become live. The old version is kept around for instantaneous rollbacks until a final approval is issued, or a given amount of time has elapsed.

Application Updates

Let’s say a company has a running application with version V1 – this is the “blue” instance. When it’s time to deploy the next application version, V2, the company deploys without exposing this new application instance to production traffic – this is the “green” instance. Instead of receiving production traffic, it is exposed to a dark URL that can be used for manual or programmatic validation.

The process of deploying and configuring applications can be elaborate. Typically, teams organize the process using steps in runbooks. Execution times vary from a few minutes to a few hours.

The team tests each step without interrupting the blue application, eliminating any downtime. The blue application continues to serve customers until the green application is ready to go.

Next, a QA pipeline runs automated tests to validate the green version. The team can triage any issues without the stress of a ticking clock. Post validation, the DevOps team or an automated tool configures the load balancers to route new requests to the green version. Once all traffic has been redirected to the green application, the blue application can be repurposed, torn down, or kept alive for a quick rollback.

A blue-green deployment in Project Borealis

This process can readily be applied to stateful applications as well. However, if your application has a database, there is additional thinking to be done about how to handle roll forward and backward compatibility.

Application Rollbacks

Blue-green deployments significantly reduce the costs associated with a broken release. If a defect is detected during or after the deployment, the DevOps team can configure the load balancer to direct traffic to the still-running blue environment, as convenient as flipping a switch. Even if your application takes significant time to start, rollback is instant since the old version is still running.

Blue-green deployments are more complicated than downtime deployments. Companies must support these environments with release automation, testing, infrastructure provisioning, configuration management, and the associated CI/CD practices. Because the deployment logic is also significantly more complicated, such strategies are often impractical to adopt without leveraging a production-ready deployment solution such as Project Borealis.

The development teams must also invest the time to make these practices work for all teams that must collaborate to successfully ship software. Often, this requires some tooling standardization. While a custom-made continuous delivery process may work for some teams, it can be challenging to apply it to all teams in an enterprise. It can also be challenging to support and maintain an extensive, homegrown solution. For example, a new engineer may not find it easy to figure out how to use all the tools available for deployment, even with simple changes. Or if a connected tool or service is upgraded, it may require troubleshooting or code rewrites before it can work.

Armory Delivers Continuous Delivery

Armory provides enterprise-grade cloud-native continuous delivery, the next evolution of the open-source Spinnaker platform. With Armory, teams can accelerate the adoption of a blue-green model using:

Pipeline building based on Spinnaker
Role-based access control
Deployment compliance checks using policies
Infrastructure provisioning using Terraform
GitOps, secret management, and more

These features enable collaboration between the developer, QA, DevOps, and SecOps teams for building end-to-end deployment pipelines. New deployment pipelines can be executed in different phases like code tagging, application release, tests, security checks, deployment, and smoke tests. This results in a green environment being set up without all the usual manual work.

Armory provides a battle-tested, flexible pipeline management system with integrated CI tools, cloud providers, security policies, and third-party libraries. Pipelines can be fully automated or user interactions can be set up representing complex and customized delivery workflows to control the pipeline. Armory enables the creation of pipelines with stages to deliver software in support of any process or policy an organization requires. Teams can go from fully automated to manual checks at each gate.

In addition to Spinnaker, Armory recently announced a new intelligent deployment-as-a-service offering, Project Borealis, that supports blue-green strategies so that developers can focus on building great code rather than deploying it. My team recently added blue-green deployments to this SaaS offering and is helping our design partners to leverage it. A demo of blue-green in Project Borealis is available. If you’d like to try it out, spots are still available for additional design partners.

Next Steps

The blue-green deployment model can eliminate the risks and toil of application release. To reap the model’s benefits, different enterprise teams must collaborate closely and have a high level of readiness.

Armory provides a foundation for safe, reliable deployments using release and deployment pipelines with compliance checks. Its policy engine and external integrations reduce the deployment model’s overall complexity, enabling development teams to operate at peak efficiency.

Are you ready to reduce your downtime and keep your application users happy while saving deployment time and effort? Explore armory.io to learn how customers use the product to solve these deployment challenges and contact us for a demo and whitepaper to see how Armory can help you. If you’d like to try out Project Borealis, our new intelligent deployment as-a-service solution that makes advanced deployment strategies simple to adopt, you can sign up to learn more and become a design partner.

The post Reducing Downtime Risk with Blue-Green Deployments appeared first on Armory.

Moving at Lightning Speed: Kubernetes Meets Blitzscaling

Thomas A. McGonagle — Fri, 11 Mar 2022 16:25:45 +0000

The wisdom of Microsoft CEO Satya Nadella’s 2019 statement said, “every company is now a software company” becomes clearer every day as more businesses seek to remake themselves around data and the digital tools to manage it.
International Data Corporation (IDC) says 53% of large organizations now have a digital transformation strategy in place and predicts half of all companies will derive more than 40% of revenue from digital products and services by 2023, up by one-third today.
That’s going to require an awful lot of software. And, most companies will tell you that building software isn’t their greatest strength. However, it’s a competency they will have to develop if they want to realize the potential to remake their businesses.
Consider that eight of the world’s 10 most highly valued companies are technology firms that distinguish themselves primarily by the innovation in their software. Even old-line heavy industries see their future there. General Motors’ Ultifi, an operating system for its built-in vehicle intelligence platform, that is expected to ultimately make thousands of software apps available to drivers, is part of what the automaker said is a wholesale transformation of its business model from carmaker to software provider.
“Today, cars are enabled by software; with Ultifi, cars will be defined by it,” said Scott Miller, GM's vice president of software-defined vehicles.
Successful companies today are moving software to market at high speed and large scale in a process known as “blitzscaling” a term coined by Reid Hoffman and Chris Yeh in their book of the same name. Blitzscaling prioritizes speed over efficiency and describes the way successful entrepreneurial firms put growth above all other considerations with the expectation that most problems can be fixed later. There’s sound business logic behind a strategy: McKinsey reports that companies in the top quartile of its Developer Velocity Index – a metric that assesses both speed and software quality – outperform others in the market by a factor of at least four.
Kubernetes enables continuous delivery
Kubernetes, which is effectively a distributed operating system for the cloud, is rapidly becoming the platform of choice for deploying software in containers. Nearly 90% of DevOps teams use Kubernetes and more than two-thirds of enterprise IT organizations will migrate their big data applications to containers managed by Kubernetes by the end of this year, according to a new Pepperdata report.
Kubernetes is an ideal platform for blitzscaling. Containers stored in libraries can be spun up in seconds, making developers instantly productive. Kubernetes scales infinitely and can orchestrate large and sophisticated applications built on a mesh of containerized services, monitoring container health and restarting services as needed. This makes it possible to extend and improve applications while they’re running.
For all its rich functionality, though, Kubernetes is not simple to install or manage. As organizations grow their cluster deployments from a few instances to hundreds or thousands, they often find they lack the skills they need to monitor and troubleshoot complex environments. Scaling the environment can also be frustrated by internal issues, such as siloed teams, homegrown tools and manual processes, not to mention the complexity of accommodating each developer’s preferred tools. Performance and account management challenges can also arise from the need to continuously retrieve the data needed to manage Kubernetes clusters.
Spinnaker can be a developer’s best friend
That’s why Spinnaker was invented. It’s an open-source, continuous delivery platform for managing software development pipelines at speed at scale. Spinnaker combines pipeline management and integrations with the major cloud providers with a host of nice features for access control, testing, and fine-tuning of deployments.
One of the most powerful features of Spinnaker is Clouddriver, a service that can create server groups, change load balancers and keep track of the health of clusters in cloud Kubernetes instances. Clouddriver’s caching agents continually monitor cloud infrastructure and log the results in a cache to enable rapid response and automated resource management.
Organizations can use the combination of Spinnaker and Kubernetes to scale to millions of clusters, but that can introduce new problems in the sheer volume of data that must flow into the Clouddriver cache. Continuous retrieval of data — even when nothing has changed — can bog down performance. If the infrastructure isn’t designed to parse the volume of data generated by large environments, the development organization can effectively be blind to the health of the deployment environment.
Spinnaker also has some shortcomings in the area of security. Its centralized architecture can make it difficult for teams to manage credentials and permissions on a per-cluster basis, a problem that becomes complex with scale. Spinnaker also doesn’t offer built-in secrets management, relying on third-party integrations for that capability.
The Armory Agent difference
Armory was built to enhance Spinnaker with the features enterprise development organizations need. The Armory Agent is an in-cluster proxy that listens for changes at the cluster level and streams them back to Spinnaker. It uses a very efficient protocol running over a single TCP connection for each cluster to reduce performance hits due to data volumes significantly. The agent leverages the Kubernetes API to enable Clouddriver to communicate to all the clusters it manages. Any changes to the infrastructure appear in the Spinnaker cache in real time. This cluster-level approach further reduces the amount of redundant data that needs to be retrieved.
Armory provides a collaborative delivery environment making software delivery continuous, collaborative, scalable and safe. Continuous delivery to kubernetes clusters at blitzscale benefit from Armory's Armory Agent, so your company can achieve blitzscale and scale to thousands of clusters on-demand without compromising security, usability or speed.
On the security front, account management is certified using in-cluster agents. This allows teams to manage credentials and permissions anytime, anywhere without restraining Spinnaker. Account management is decentralized, and Kubernetes API servers can be kept private from Spinnaker.
The result is that individual developers can get a single and holistic view of deployments and application infrastructure that extends from their local laptops into production environments. Armory abstracts away much of the complexity to let them take ownership of their services without having to become infrastructure experts. Development organizations can use Armory to scale to thousands of clusters or more on-demand without compromising security, usability, or speed.
That sounds a lot like blitzscaling, doesn’t it?

Armory Agent: Managing Kubernetes at Scale

danielGz — Thu, 10 Mar 2022 17:06:31 +0000

Almost half of all organizations now use Kubernetes (K8s) to empower continuous integration and continuous development (CI/CD) and stay competitive in a software-centric world.

The challenge? Effective management becomes an issue as K8s deployments scale from tens or hundreds of clusters to thousands or more. From lacking skills to existing integration issues and legacy incompatibility, many organizations encounter roadblocks when they move from initial deployments to more robust Kubernetes frameworks.

Armory is leading the charge on improved Kubernetes management — first with Spinnaker and now with the availability of the Armory Agent. This lightweight, scalable service monitors your K8s deployment and streams changes back to Spinnaker’s Clouddriver service.

But what’s under the hood? Let’s explore how the Armory Agent can help streamline Kubernetes at scale.

Connecting with Kubernetes

As developers scale Kubernetes clusters exponentially, they often run into corresponding performance and account management challenges. On the performance side, continuous retrieval of data — even when nothing has changed — can negatively impact operational efficacy. These recent polls from Pulse, show just a few of the challenges:

On the account management side, the centralized nature of Spinnaker makes it difficult for teams to manage credentials and permissions on a per-cluster basis.

The Armory Agent offers a new approach to Kubernetes and Spinnaker management designed to facilitate connections at scale.

Exploring the Armory Agent Advantage

The Armory Agent listens to changes at a cluster level and then streams them back to Spinnaker. As a result, any changes to your infrastructure — whether initiated by Spinnaker or not — appear in real-time in the Spinnaker cache over a single TCP connection per cluster. This cluster-level approach helps reduce the amount of redundant data accessed and retrieved, boosting overall performance.

On the account management side, Armory Agent allows teams to manage credentials and permissions anytime, anywhere without restraining Spinnaker.

Management, Evolved

Bottom line: Kubernetes deployments backed by Spinnaker infrastructure are now essential for effective CI/CD. But there’s always room for improvement, especially when it comes to scalability. By integrating the Armory Agent, organizations can scale to tens, hundreds, or thousands of clusters on-demand without compromising security, usability, or speed.

The post Armory Agent: Managing Kubernetes at Scale appeared first on Armory.