Forem: danielGz

Managing Compliance with Continuous Delivery

danielGz — Tue, 05 Apr 2022 15:52:49 +0000

This article explores how continuous delivery helps organizations maintain compliance and discusses how Armory helps with features such as rollbacks and progressive deployments to tighten control over your solution infrastructure. It shows how to improve code deployments, ensure compliance, and maintain a healthy and active innovation ecosystem between your developers and operations team.

Developers and DevOps teams are often stuck with highly manual, error-prone, and resource-intensive processes that do not always meet the requirements of various laws and regulations. Every industry has its own set of regulations and compliance rules. For example, the finance industry has extensive regulatory agencies and regulations, such as:

The Federal Risk and Authorization Management Program (FedRAMP), which applies to companies in the cloud
The Payment Card Industry Data Security Standard (PCI DSS), which regulates companies accepting online payments
Service Organization Control (SOC2), which helps establish trust in and transparency into an organization’s service delivery processes and controls

Additionally, some general regulations, such as the European Union’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), apply to organizations serving a regional or international audience. Organizations must vet every software iteration for compliance with these regulations before deploying fully.

Constantly changing regulations in industries like finance are especially challenging for organizations using a monolith system of application development and deployment. They can’t quickly meet changing compliance requirements and may face delays changing part of an application, which slows innovation. Monolith application delivery becomes challenging because you must run compliance tests on the entire tech stack for every change.

Many organizations turn away from monolithic applications and embrace continuous integration and continuous delivery (CI/CD) to meet regulatory and other challenges. Modern CD tools help technical teams quickly, and sometimes automatically, roll out changes to meet shifting regulatory requirements while maintaining audit logs.

Let’s explore how Armory improves your CD pipeline reliability and flexibility using progressive deployment strategies with one-click rollbacks to minimize risk. Additionally, we’ll discuss how automated policy enforcement ensures your team runs scans and follows configuration best practices.

Frequent Updates Using Continuous Delivery

Organizations are moving toward the microservices-oriented development and deployment model. This development model helps their engineering teams focus on individual challenges inside the entire solution and solve them quickly. Since each team maintains a single code base, it’s easier to regulate and ensure compliance. Also, in case of any problem, the team can easily roll back their deployment to a previously compliant state.

A typical enterprise application might comprise hundreds of small processes called microservices. Validating the compliance and regulation checks on hundreds of different applications is more manageable than one extensive application. This is because you can easily pin and regulate a non-compliant process during deployment checks. If a microservice isn’t compliant, the team rejects the deployment for that microservice only — not the entire stack. This rejection also alerts the developers responsible for the microservice’s maintenance to ensure compliance in their codebase.

Sometimes it’s not technically possible to debug and run the solution locally. For example, if your teams must provision and analyze the logs your app generates, it might not be feasible to run the entire cluster on a developer machine. However, provisioning a test or development environment for every team is expensive in licensing, hardware, and staffing. In contrast, with microservices, each team can run their project locally, ensure compliance, then push it for deployment. Your IT teams can run their deployment checks on each release to either approve or reject the ones needing improvements.

After you’ve passed your compliance checks, progressive rollouts work to minimize the impact of any undetected issues. Progressive rollouts direct a subset of traffic to the microservice’s new version while most traffic continues using the old version. This allows you to inspect the new version with a subset of the production workload, minimizing the impact of any undetected issues. When the versions have run in parallel without issue, you can scale up the new version and reroute the remaining traffic from the old version. If you discover an issue at any point in the deployment, one-click rollback enables you to reroute traffic back to the old version.

Continuous delivery ensures your IT and compliance teams are on top of every release to your infrastructure. In the following sections, you will learn how Armory can help them with production deployments, as well as with one-click rollbacks and progressive deployments. Although companies mainly use microservices to test and verify new functions in isolation, minimizing their footprint and decreasing software delivery risks, microservices also help your teams quickly apply changes as governing bodies introduce new regulations.

Flexibility and Reliability

Maintaining a continuous delivery pipeline can be challenging as you manage smooth rollouts and rollbacks. Continuous delivery requires you to test and verify your deployments for functionality and compliance. When your deployment doesn’t meet the production service-level agreement (SLA), you should be able to revert the changes to the previous working state.

Progressive deployments and one-click rollbacks help you control your DevOps pipeline. The progressive deployment makes sure your app rolls out to the targeted audience that you define. This function helps your IT teams gradually allow a group of customers to access your latest deployment, ensure everything is good to go, and then provide the newest deployment progressively to the rest of your audience.

The benefit of this staggered release is that you can control and quickly revert the application to a working state if your platform crashes or you find any bug. Armory’s one-click rollback, for example, helps you revert the recent changes to your infrastructure, such as Kubernetes.

With Armory, you can control when a rollback happens. This helps you prevent unnecessary rollbacks if something else in your production breaks. Armory is the enterprise-scale version of Netflix’s open-source software Spinnaker, and its features make your continuous deployments easier. Spinnaker supports major public cloud vendors, so you can easily connect Armory to any public cloud and start deploying your production stack.

When you find a compliance issue, you can revert your app to a compliant state using rollbacks. Rollbacks help you avoid inadvertently breaking the rules and incurring fines, losing a license, or other regulatory actions.

Controlling Continuous Delivery

The goal of continuous delivery is to prepare every code commit for deployment. Apart from verifying application functionality, you also need to run specific policy validations on your deployments to ensure security and compliance. The Armory Enterprise Policy Engine can run policy checks on your deployments to ensure security and compliance. Armory’s example policies can get you started.

You can also extend Policy Engine by writing custom policies using Open Policy Agent. These policies ensure compliance and make it easier for your compliance and IT teams to detect red flags in deployments. Your organization can write custom policies to ensure automated test suites and security scanners run before deploying your code to production. These policies include, but are not limited to, security bugs, hardcoded passwords or connection endpoints, logging user data, opening unnecessary ports, and more.

With this custom policy feature, you can control which images are allowed to deploy to production and which the tool should reject.

In addition to enforcing policies around your software delivery pipeline, Policy Engine can also enforce most policies that have traditionally been implemented with Kubernetes Gatekeeper. This decreases Kubernetes management overhead by enabling you to enforce your policies in a single place instead of independently in each Kubernetes cluster.

Some organizations don’t allow developers to access certain production logs. This lack of access slows down the deployment process, and when there is a production problem, the developer has no idea of what is going wrong. Armory with Spinnaker helps you maintain deployment logs while ensuring it doesn’t write sensitive information anywhere. You can then provide these logs to external or internal auditors.

Conclusion

In this article, we discussed the challenges apps face with compliance and regulations. Monolith app development and deployment are more likely to slow app deployments and releases by making it challenging to run compliance, quality, and regulation checks on deployments, slowing the entire DevOps pipeline, and making it challenging to roll back infrastructure changes. Continuous delivery helps overcome these challenges.

You learned how Armory’s DevOps and continuous delivery features, such as progressive deployments and rollbacks, help your IT team remain on top of every deployment. Controlling every deployment helps quickly catch any compliance issues before reaching your audience (and the attention of regulators).

The post Managing Compliance with Continuous Delivery appeared first on Armory.

Armory Agent: Managing Kubernetes at Scale

danielGz — Thu, 10 Mar 2022 17:06:31 +0000

Almost half of all organizations now use Kubernetes (K8s) to empower continuous integration and continuous development (CI/CD) and stay competitive in a software-centric world.

The challenge? Effective management becomes an issue as K8s deployments scale from tens or hundreds of clusters to thousands or more. From lacking skills to existing integration issues and legacy incompatibility, many organizations encounter roadblocks when they move from initial deployments to more robust Kubernetes frameworks.

Armory is leading the charge on improved Kubernetes management — first with Spinnaker and now with the availability of the Armory Agent. This lightweight, scalable service monitors your K8s deployment and streams changes back to Spinnaker’s Clouddriver service.

But what’s under the hood? Let’s explore how the Armory Agent can help streamline Kubernetes at scale.

Connecting with Kubernetes

As developers scale Kubernetes clusters exponentially, they often run into corresponding performance and account management challenges. On the performance side, continuous retrieval of data — even when nothing has changed — can negatively impact operational efficacy. These recent polls from Pulse, show just a few of the challenges:

On the account management side, the centralized nature of Spinnaker makes it difficult for teams to manage credentials and permissions on a per-cluster basis.

The Armory Agent offers a new approach to Kubernetes and Spinnaker management designed to facilitate connections at scale.

Exploring the Armory Agent Advantage

The Armory Agent listens to changes at a cluster level and then streams them back to Spinnaker. As a result, any changes to your infrastructure — whether initiated by Spinnaker or not — appear in real-time in the Spinnaker cache over a single TCP connection per cluster. This cluster-level approach helps reduce the amount of redundant data accessed and retrieved, boosting overall performance.

On the account management side, Armory Agent allows teams to manage credentials and permissions anytime, anywhere without restraining Spinnaker.

Management, Evolved

Bottom line: Kubernetes deployments backed by Spinnaker infrastructure are now essential for effective CI/CD. But there’s always room for improvement, especially when it comes to scalability. By integrating the Armory Agent, organizations can scale to tens, hundreds, or thousands of clusters on-demand without compromising security, usability, or speed.

The post Armory Agent: Managing Kubernetes at Scale appeared first on Armory.

Resiliency and Load distribution

danielGz — Thu, 16 Dec 2021 20:39:40 +0000

Introduction

When scaling a network service, there are always two concerns: resiliency and load distribution, to understand these concepts let us first understand the broader term “Redundancy”.

Redundancy is the duplication of a component to increase reliability of the system, usually in the form of a backup, fail-safe, or to improve actual system performance.

Resiliency is part of the “reliability” goal of redundancy, it comes in the form of High Availability (but it’s not limited to this).

Load distribution is part of the “system performance” goal.

Let’s not forget that there is also a case for redundancy when you want to have a service available in different zones, i.e clients in a different part of the world not having to send requests to a service hosted in a different continent, but this is a special case of replication for performance from the client’s perspective (although could be used for resiliency as well). We will leave this case out for now.

Why does resiliency matter?

In an ideal case, we want to optimize for both resiliency and load but achieving this is challenging as there are always trade-offs and things to take into account when deciding how to do it.

Let’s dig deeper into this with a real-world scenario.

At Armory, we focus on deployment solutions at scale. Armory Agent for Kubernetes(Agent for short) is able to monitor Kubernetes clusters and execute CRUD operations of Kubernetes objects in it. It works as a bridge between Kubernetes and a continuous-delivery tool such as Spinnaker.

As you can imagine, any pipeline that targets a Kubernetes cluster will rely on the Agent. If the service is not available, there is a risk of pipelines halting. Much worse, you can end with an unexpected result the next time you retry it if the pipeline is not idempotent.

It is not realistic to expect atomicity in a pipeline or side-effect-free pipelines on a continuous delivery tool. This is where Resiliency comes into play. In the case of Armory Agent, having multiple replicas to ensure that the service is highly available will help prevent scenarios like the above.

How is high availability achieved in software engineering?

High availability is achieved by replicating a component. And you get exact replicas by copying the state across replicas or by making a service as stateless as possible. That’s part of the popularity of JWT over sessions in REST: a system won’t have to migrate a session if a request arrives in a different replica other than the original one because a REST API is able to operate in a stateless manner.

Now going back to our case, remember that one of the features of the Agent is monitoring Kubernetes clusters.

What happens now is that you have more than one service monitoring the same cluster(s). While this is not a problem itself, nothing comes for free. In software engineering, there is no magic behind anything. Any extra events being sent means extra network calls, extra CPU usage and extra RAM usage of a node or server for events that the original replica is already sending.

Granted if one instance goes down, you can rest assured knowing that an identical one is hot and ready. However, there is not much use in monitoring the same cluster in different replicas and having all of them report the same stuff.

So how do we attain load distribution?

We have two main options:

Configure each instance differently but you no longer have fault tolerance because now each of them become critical to their own subset of tasks.
Apply a mechanism to make each replica aware of the other. Or you can have an orchestrator of replicas, but you introduce statefulness in the later cases.

But what happens if a master replica is getting clogged with too many write requests? Or what happens if the orchestrator goes down? For these issues, there are no other options than replicating the orchestrator or the master replica itself.

Does that mean it is not possible to achieve load distribution and high availability at the same time?

No, it does not. It is possible but there is cost involved depending on the type of service you are scaling. A REST API that’s doing nothing but waiting for HTTP requests can 100% be scaled and achieve both high availability and load distribution with the same instances. On the other hand, a stateful component, such as a database, would need a copy of its state across replicas and a master node as well as an active component, such as a message broker or Armory’s agent for Kubernetes who are constantly pushing events need an orchestration of some kind. These are a few examples:

RabbitMQ : needs an additional message exchange with the sharding plugin. This means having one of the nodes perform additional tasks to maintain this.
Armory Agent : requires configuring each instance differently or having account segmentation upon registration by the client (such as in the Clouddriver service).
Apache Kafka: requires having different partitions in each broker, and there is usually a partition leader who resolves to which partition each message goes to based on its key. This means that you have additional work made by one broker and also one broker could be storing more data than others. And then all of this state is kept in Zookeeper.

What should come first, scalability or high availability?

High Availability should come first due to the risk of not having it, especially if there is loss of data involved. Usually scalability comes afterwards to support the first. There is no sense in having a highly performant cluster of services if it means that one of them crashing will crash the whole system.

Conclusion

Replicating for performance and fault tolerance are two different things, and you can’t expect to be able to tackle both with the same group of replicas.

When trying to achieve robustness in a software environment, it is very important to be aware of the nature of the components to scale, and scale accordingly depending on your use case.

You may find that a balance between the both is the way to go but never overlook the high availability of your services.

References

https://docs.armory.io/ae/armory-agent/

https://docs.oracle.com/database/121/REPLN/repmaster.htm#REPLN002

https://www.confluent.io/blog/scaling-kafka-to-10-gb-per-second-in-confluent-cloud/

https://cndoc.github.io/redis-doc-cn/cn/topics/sentinel.html

https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_sharding

The post Resiliency and Load distribution appeared first on Armory.

The Paperboy...

danielGz — Tue, 07 Dec 2021 23:18:07 +0000

Its 5AM, its pitch black out, and its 34 degrees and sleeting. The wind is screaming, you’re 9 years old, and you have to deliver the papers. You get up, put on 3 pairs of sweat pants, and 4 sweatshirts. 60 lbs of papers have just been dropped off, and you grab your flashlight and a razor to open the bindle. You spend the next 30 minutes in your garage folding the papers into tight packets and you load up your paper route bags which are tied to your bike.

You head out into the snow, and walk your bike the first half of the route which has 3 monstrous hills. The bike eventually becomes light enough to ride and the cold becomes less and less horrible. It takes over 2 hours to deliver a paper to each of your 160 customers. Barely enough time to get ready and get to school.

This was my daily ritual 7 days a week for 3 years delivering the Boston Globe in Winchester, MA. The job was hard but lucrative and it taught me everything I ever needed to know about business. My paper service provided a $5,000 dollar college scholarship which you were entitled to for three years of service. $5,000 in the late early 90s was a lot of money for college. My undergrad degree from Bentley University was $75,000 and I minimized that further by receiving grants to play Division II Football as an Offensive Left Tackle, and working 20 hours a week at a Dotcom startup, e-Dialog, where I discovered Linux in 1997, and sent me on the trajectory I find myself on today at Armory.io.

Delivering the Globe taught me the value of hard work and how life is linearly rewarding. The more you put into something the more you get out of it. Anything worth doing is hard, and strength can be manifested in character and body. My paper proceeds were invested in the nicest X-men comic book collection in the North East.

At times I felt like Sisyphus. My 80 pound bike was a boulder and I was a bit of a trickster who in fact had cheated death. I spent the better part of my 3rd year in Children’s Hospital with a wicked infection in my right ankle which massive amounts of antibiotics could barely control.

Both of my parents had paper routes, and I was always a business man. My paternal grandfather Daniel R. McGonagle Sr. put three kids through college selling flashlights door to door. He opened one door at a time and emptied his car trunk literally every day seven days a week.

I quickly discovered there was a side business opportunity for a paper boy walking dogs. I had a business called Pet Care Extradinaire, and my tagline was: If you have something to do and your pets being a pest, give me a call and Ill do all the rest.

In the afternoons I had the keys to 15-20 houses and would walk their dogs. Some were good dogs and some were bad. And I was lucky I wore 4 sweatshirts as dog attacks are a hazzard of the paper route and dog walking professions. The worst and last one was very bloody, and I always carried a can of mace, and a wrench afterwards.

Today, I value the hard work that I put into that paper route. It taught me so many valuable lessons that I still use today. I am sure there are other jobs that have done this for others and I’d love to hear about them.