Forem: Codefresh

How to Preview and Diff Your Argo CD Deployments

Kostis Kapelonis — Tue, 17 Jan 2023 14:36:25 +0000

Adopting Kubernetes has introduced several new complications on how to verify and validate all the manifests that describe your application. There are several tools out there for checking the syntax of manifests, scanning them for security issues, enforcing policies etc.

But at the most basic case one of the major challenges is to actually understand what each change means for your application (and optionally approve/reject the pull request that contains that change).

This challenge was already present even outside GitOps, but it has become even more important for teams that use GitOps tooling (such as Argo CD) for their Kubernetes deployments.

The problem

Any major Git platform has built-in support for showing diffs between the proposed change and the current code when a Pull Request is created. In theory, the presented diff should be enough for a human to understand what the changes contain and how they will affect the target environment.

In practice however several teams have adopted a templating tool (such as Kustomize or Helm) that is responsible for rendering the actual Kubernetes manifests for a target cluster.

As a quick example let’s say that you need to review a Pull Request with the following changes:

This seems simple enough. You assume that this change will increase the number of replicas to 20 (let’s say it is Black Friday and you want to increase the capacity of your web store ASAP). You merge the pull request and … nothing happens.

What you didn’t know is that there is a downstream Kustomize overlay that also defines replicas on its own. So the proposed change has no effect at all. The problem was that the pull request contains only the segment of a Kustomize source manifest and doesn’t show a diff for the end result (the full rendered manifest).

The problem is even more apparent when your organization is using Helm. Let’s say that you need to approve a pull request with the following changes:

As a human, it is very difficult to understand what exactly is happening here. You need to mentally run the templates in your head and decide if this change is correct or not. Wouldn’t it be nice if the diff had the actual manifest that is created from this chart?

Essentially the diff functionality found in your Git system is not enough when it comes to complex Kubernetes applications.

Using the built-in Diff functionality in the Argo CD GUI

One of the main benefits of using the Argo CD UI during a deployment is the built-in diff feature. When a resource is “out-of-sync” (i.e. it differs from what is in git) Argo CD will mark it with a special color/icon. In the following example, somebody has changed the service resource of an application:

You can then click on the service and see the diff

The big advantage here is that Argo CD has already integrated support for Kustomize and Helm. The diff you will see is on the final rendered manifests which is exactly what you want as you can preview changes in their full context.

Unfortunately, this method also has several disadvantages.

The first one is that Argo CD shows only diffs for applications when the auto-sync (and self-heal) behavior are disabled. This means that you are losing the main benefit of GitOps. The proper way to follow GitOps is to have auto-sync enabled (and self-heal as well) as this way guarantees the basic premise that the cluster and the Git repository contain the same thing, that is, that the desired state and actual state have not diverged.

But the second problem regarding Continuous Delivery is that the diff on the manifests is shown when the changes are already committed and pushed. And this is too late to perform any serious review. Ideally you want to review changes as early as possible. A Pull Request allows you to add comments, talk with your team about the changes and also reject the Pull Request altogether without affecting a production system.

Using the built-in diff functionality in the Argo CD GUI is great for validating a change and doing a last sanity check just before production. But it should not be the main review milestone of a manifest change. And ideally you should setup all your applications to sync automatically, so having this diff process is not available in the first place.

Using the local Diff feature of the Argo CD CLI

We have seen the built-in diff UI in Argo CD is shown very late in the delivery process. Can we use the same diff approach earlier in the life of a change?

It turns out that the Argo CD CLI also comes with a diff command. This diff command takes a “–local” parameter that allows you to compare what is happening in the cluster against ANY local files which don’t have to be pushed (or even committed at all). It will also automatically run your favorite template tool as it is defined in the Argo CD application.

Here is how it looks

This approach is very promising as you could in theory use it inside a CI system with the following process:

Open a pull request with the suggested changes
Have your CI system checkout the pull request
Run inside a CI pipeline argocd diff –local against the cluster where the pull request is destined (It also uses again the built-in support for kustomize/Helm within Argo CD)
Present the diff to the user in order to take decisions about the pull request

This sounds great in theory but in practice has several shortcomings.

The most obvious one is that you need to provide your CI pipeline with credentials that access the cluster where Argo CD is installed. This forfeits one of the main benefits of GitOps – the pull mechanism where the credentials stay within the cluster.

An even bigger concern however is what happens when you have multiple clusters. Which cluster should you pick to compare against? What if the chosen cluster has CRDs or other resources that are custom to it?

This process can also become very complex with remote or secure cluster instances. For example if you have an Argo CD cluster in Asia and your CI system is running in the US, connectivity between the two might be very slow or even impossible.

In summary “argocd diff –local” is great for local experimentation and quick adhoc checks, but for a production deployment process there is a better way to achieve the same result (spoiler: it doesn’t involve the Argo CD CLI at all neither needs cluster access).

Pre-rendering manifests in a second Git repository

Let’s take a step back. We have been looking for ways to show an enhanced diff as part of a pull request and ignore the existing diff that is already provided by the Git provider (as we have seen that this doesn’t work with the final manifest).

There is a way however to enhance the built-in diff and make it work on the final manifests.

The solution is to use 2 GitOps repositories, for each application/cluster. One git repository has the manifests in their unprocessed form (e.g. as Kustomize overlays) as before. There is now a second Git repository that has the final rendered manifests. And Argo CD is pointed at the latter.

Here is how it would look:

This process should be familiar to you if you have ever used a preprocessor or code generator. Essentially there is an automated process (can be the CI system or something else) that does the following:

A human creates a pull request on “Source” Git repo with the suggested change
A “copy” process takes the contents of the pull request and applies the respective template tool (i.e. Helm/Kustomize) to create the final rendered manifest
A second pull request is opened automatically on the “Rendered” Git repo with the contents of the manifests
A human sees the diff of the second pull request and this time the diff is between rendered manifests and not snippets/segments.
If the Pull Request is approved it is merged on both Git repositories. Thus the second repository has always rendered manifests
Argo CD monitors the second repository and applies the changes (the integrated support for Helm and Kustomize within Argo CD itself is not used at all, Argo CD is only syncing raw manifests)

This is a very valid process and I have seen it used in several companies with success.

The big advantage of course is that the diff you get in the Git provider provides you with the full information about what will change in the application AFTER all manifests are processed:

Here is the same example with the Helm chart, but this time we are using a second Git repository that has the rendered manifest stored.

However, I am personally against this process, as it complicates things a lot and increases the number of moving parts.

It doubles the number of repositories for any given application (or branches if multiple branches are used)
It introduces another point of failure which is the copy process that converts source YAML to final manifests
It completely bypasses the effort put in Argo CD to process manifests on its own.
It might be confusing for people who now have 2 Git repositories to work with and opens the possibility of mistakes in both ends (either committing stuff on the “source” repo that never makes it to “rendered” repo or vice versa).

In general I think that this is an overkill solution for a problem that can be solved more elegantly as we will see later in the article. Still, if you follow this approach and it works for you, make sure that you have safeguards and monitoring in place (especially for the copy/commit automated process).

Intermission: Preview Terraform plans

You might think that previewing the full manifests for a pull request is a new problem that Kubernetes introduced. It isn’t. There have been several tools before Kubernetes that had to deal with the exact same issue and it would make sense to look at what they do.

The most obvious candidate to examine is terraform. If you are not familiar with terraform it is a declarative tool that allows you to define your infrastructure in a HCL file and then “apply” your changes.

Terraform users have a very similar problem. A pull request that contains terraform changes (especially in big projects) is not immediately clear for a human to understand (unless you are an expert on running terraform mentally in your head).

To solve this issue, terraform has a “plan” command which reads the changes, decides what it will do and prints a nice summary of all the proposed changes without actually doing anything.

The plan functionality in terraform is crucial to terraform teams as it removes the guesswork on what terraform will do when the changes are applied.

With this summary at hand, the next step is obvious. We can simply attach the output of the plan command to the pull request. Humans can now look at both the diff of the hcl files but also the plan summary and decide if the change is valid or not.

This workflow is so common that an open source project – https://www.runatlantis.io/ does exactly that.

You make your changes on the terraform files
You create a pull request
Atlantis runs the “plan” command and attaches the result in the PR
You can then approve/comment the PR as a human
Atlantis then runs the “apply” command to actually modify your infrastructure
Atlantis locks the workspace until the PR is merged, preventing a second PR from overriding the changes before the first PR is merged.

This workflow is very effective for end-users but has several security drawbacks which are similar to the argo CLI diff approach

You need bidirectional communication between your Git provider and the server that runs Atlantis
The server that runs Atlantis will run terraform on its own and thus it needs all credentials that terraform has in your organization.
The server that runs Atlantis also needs to have access to your remote terraform state. Essentially Atlantis has the keys to your kingdom.

Another downside of Atlantis is that it’s Pull Request based, whereas proprietary Terraform CD tools on the market feature a dashboard where every project / workspace can be browsed (similar to how you have a dashboard in ArgoCD where you can see your Applications and whether or not they are synced).

In theory we could create a similar system for argocd diff –local but given the security implications, there is a much better approach that is helped by the GitOps principles.

Still, the basic idea of attaching a diff in a pull request for greater context offers several advantages for the end user that cannot be overstated.

Render Kubernetes manifests on the fly

One of the most important principles of GitOps is that at any given time the cluster state is the same as what is described in the git state. We really like how terraform Atlantis works but there is a way to improve the workflow and make it more secure and more robust by taking advantage of the Argo CD guarantee for GitOps.

The Terraform CLI that runs on the Atlantis instance needs credentials to your infrastructure because it must both read the terraform state and also create the actual infrastructure once changes are “applied”.

With Argo CD we can completely bypass this limitation because we already have the cluster state right there. It is stored in the target of the Pull Request!

This means that we don’t need any credentials either to the cluster or to Argo CD. We can simply run a diff between the files of the Pull request and the branch it is targeted at. The extra addition here is that we will also run a preprocessing step for the template solution (Helm or Kustomize) in order to get the full manifest.

So the full process is as follows:

Somebody opens a Pull request to the manifest repo
We check out the code of the pull request and run Kustomize, Helm or other templating tool in order to have the final rendered manifests of what is changed
We check out the code of the branch that is targeted by the Pull Request (e.g. main) and find the same environment and again run the same templating tool to get the final manifest
We run a diff between the final manifests from the two previous steps
We show the diff to the human operator that will decide if the pull request will be merged or not

The beauty of this approach is that unlike Atlantis, we never access the Argo CD cluster. All information is coming from Git (an advantage of using GitOps). This means that your Argo CD cluster could be in China with a very slow connection (or even an isolated connection) and your CI server doesn’t need to know anything about it. In fact the location and security access of your Argo CD server is now irrelevant as we don’t interact with it in any way.

Notice in the diagram above that unlike Atlantis, our CI server has a direct connection only to the Git repository. The Argo CD instance still cares only about the Git repository it monitors.

This makes our approach much more secure as the credentials of the cluster still stay within the cluster itself and we only interact with the Git repository.

One thing to notice here is that unlike Atlantis or argocd diff command we are not comparing desired state in Git to actual state (acquired from cloud provider’s API or Kubernetes API), we are comparing two versions of desired state stored in different branches of a Git repository. While being a good enough approximation, this approach is not 100% equivalent to the argocd diff one.

A corner case scenario would be Helm Capabilities – built-in variables populated by querying k8s cluster for API version and available resources. Some Helm templates use this information to render correct resource versions, appropriate for specific cluster’s version and available CRDs. This information has to be supplied manually to “helm template” command to achieve parity with argocd diff.

Attaching the full manifest diff to a pull request

The icing on the cake is that we will also attach the full manifest diff to the Pull request (as Atlantis does).. This is how it would look:

We now need to explain to users that the automatic diff of the pull request is not what they should look at anymore because it only has part of the story (see the problem description at the beginning of this article). Instead, they should look at our attached diff and get the full picture of what has changed and make decisions accordingly.

The attached diff is especially important for people that use Helm as you can see a diff between plain YAML instead of trying to run manually Golang templates in your head.

Enforcing changes during environment promotion

If you follow this diff approach where the full manifests changes are shown in the Pull Request it will be much easier for you and your team to collaborate on GitOps changes as everybody will have the full context of each incoming change.

A secondary benefit however of this diff approach is also to know what is NOT changed.

In my previous article about promotions between GitOps environments using folders, a lot of people asked about how you can guarantee that extracting a common setting from downstream Kustomize overlays and promoting it your base overlay can be safely executed in a single step.

I was really puzzled by this query, until I realized that most people that were asking this question looked at the simpler diff of the pull request and thus lacked the full context of the change.

Let’s take an example. You have two environments qa and staging with the following settings:

UI_THEME=light
CACHE_SIZE=2048kb
SORTING=ascending
N_BUCKETS=42

You want to add a new setting called PAGE_LIMIT=25 and promote it gradually first to qa and then to staging

You modify/commit the qa environment

UI_THEME=light
CACHE_SIZE=2048kb
PAGE_LIMIT=25
SORTING=ascending
N_BUCKETS=42

The deployment goes ok and you make the same change to the Staging environment. It works fine there as well.

Now you decide that this new setting should be the same across both environments and you decide to move it to the parent overlay (which is common to all non-prod environments).

So the actions you take are

Delete the setting from the QA environment
Delete the setting from the Staging environment
Add the setting into the parent overlay that both environments depend on
Commit/Push all the above in a single step

A lot of people were concerned about this process and asked how you can enforce that the whole process will work without affecting the existing environments.

We can finally answer this question by simply looking at the enhanced diff of the above commit

That’s right. All diffs are completely empty. Even though there are changes in the individual Kustomize files the end result (the rendered manifests) are EXACTLY the same.

This means that if you approve this Pull request Argo CD will do absolutely nothing and you are certain that all environments will be oblivious to this refactoring.

Of course the basic diff of the pull request is not that smart and shows the diff changes in text in the individual files.

So in this scenario we have the extreme case when the built-in diff of the pull request doesn’t have the full context of what is going on because it doesn’t understand the full manifests.

Conclusion

Previewing changes before applying them is a pillar of modern software automation and in the case of Kubernetes applications this is not always a straightforward process because of the templating of the manifests.

In this article we have seen several ways of previewing the changes in Argo CD applications:

Basic diff of the Git platform (not recommended)
Native diff of the Argo CD UI
Diff local files with the Argo CD CLI
Pre-rendering manifests in a second Git repository
Rendering manifests on the fly for each Pull request (recommended)

We hope that this process is helpful for you and your team when of course it is combined with static analysis, syntax validation, security scans and other sanity checks that run against your Kubernetes manifests.

Happy diffing!

How to Install and Upgrade Argo CD

Kostis Kapelonis — Tue, 17 Jan 2023 12:28:49 +0000

We have already covered several aspects of Argo CD in this blog such as best practices, cluster topologies and even application ordering, but it is always good to get back to basics and talk about installation and more importantly about maintenance.

Chances are that one of your first Argo CD installations happened with kubectl as explained in the getting started guide. While this form of installation is great for quick experimentation and for trying Argo CD, there are many more installation methods that are recommended for production deployments.

Manual installation and manual upgrade

The most obvious installation method is using the official manifests or the respective Helm chart. Note that the Helm chart for Argo CD is not official and sometimes it lags behind the regular Argo CD releases.

While the initial installation of Argo CD using just the manifests is quick and straightforward it also suffers from several shortcomings:

Configuration changes (e.g. SSO and notifications) must be also applied manually without any option for rollbacks or auditing
No disaster recovery option if your Argo CD instance goes down
Extra effort is needed for modifications on the base install (e.g. for Argo CD plugins)
It becomes very cumbersome to manage multiple Argo CD instances in a manual way

However, the biggest challenge is actually upgrading Argo CD in a safe way. New Argo CD versions come with their own notes and incompatibilities and trying to manually upgrade your instance without any backup plan is a recipe for disaster.

In summary, you should only employ manual installation via manifests for quick prototypes and demo installations.

Using a hosted Argo CD instance

If you are searching for the easiest way to use Argo CD while still having a production installation, look no further than a hosted Argo CD instance. At Codefresh, we already announced our hosted Argo CD offering earlier this year. This Argo CD instance is completely managed by Codefresh personnel. The only thing you need to do is connect your cluster for deploying your applications.

The main advantage of this method is that all maintenance effort is handled by Codefresh and not you. All version updates, security fixes and other upgrades are automatically handled behind the scenes by Codefresh and you can focus on deploying applications.

The hosted version of Argo is available to everyone that signs-up with Codefresh, including free accounts.

Using Argo CD to manage Argo CD

Using a hosted instance of Argo CD can be great for many organizations, but may not be a fit if you need to deploy behind the firewall, or need more customization. Ideally you would like to customize your Argo CD installation, setup different settings, configure your own plugins, pin down specific Argo CD versions etc

Hosting your own Argo CD instance is popular, but instead of doing it manually you should use a management platform on top of it. And the most obvious choice would be managing Argo CD with itself!

This use case is perfectly valid and a lot of organizations use self-managed Argo CD. The advantages are:

Using GitOps to handle not just applications but also the Argo CD installation
Full audit via Git
Easy rollbacks
Automatic drift detection for any manual changes
Complete changelog of all configuration changes (e.g. notifications SSO)
Easy disaster recovery

This is a great way to handle a production instance of Argo CD. Depending on the size of your organization it will still suffer however from some important challenges:

You still have to perform manual upgrades and make sure that each new version of Argo CD “sits” cleanly on top of the previous one
Handling a large number of Argo CD installations and keeping them all in sync (pun intended) is still a big challenge.

Using Argo CD Autopilot

The use case for using Argo CD to manage Argo CD is very popular as a concept but there are no best practices yet on how to get started and how to bootstrap the whole environment.

We use the same approach internally and we fully open-sourced our solution at https://argocd-autopilot.readthedocs.io/en/stable/

Argo CD autopilot provides a CLI installing and managing Argo CD that does the following:

Connects to your Git provider
Bootstraps Git repositories for handling both applications and itself
Setup Applications and ApplicationSet for auto-upgrading itself and other managed apps
Provides a best practice Git repo structure for both internal and external applications
Comes with a CLI that allows you to manage and maintain the installation
Introduces the concepts of deployment environments/projects

Argo CD Autopilot is under active development. You are welcome to participate in Github as well as the #argo-cd-autopilot channel in the CNCF slack.

Using a control plane

Handling one or two Argo CD instances is pretty straightforward if you choose any of the above installation methods. Several organizations however have a large number Argo CD instances that need to be kept in sync or rolled gradually as new versions come out.

Argo CD can natively support a management instance that handles multiple deployment clusters. So in theory you could have a single Argo CD instance for all your environments. We have already talked about this pattern in our article about scaling Argo. In the end, having a single instance is a single point of failure and also comes with its own issues for security and redundancy.

On the other hand having an Argo CD instance for each deployment cluster is also excessive and can lead to a cumbersome setup where maintaining Argo CD instances becomes tedious and unmanageable, especially across virtual private clouds and firewalls.

Ideally you would like a single management interface that can handle all possible combinations (Argo CD management cluster and deployment clusters) allowing you to craft your perfect topology.

This management interface exists in the form of the Codefresh GitOps control plane!

The Codefresh platform gives you a unified interface for handling all Argo CD instances no matter where they are located. All possible configurations are supported:

Argo CD management clusters
Argo CD deployment clusters
Hosted Argo CD installations
Deployment clusters managed by the hosted instance
Argo CD instances that deploy on the same cluster they are installed on
Argo CD instances that are deployed behind a firewall or on-premise environment

Via the control plane interface you can then

Connect Argo CD instances
Connect target deployment clusters
See the status of each Argo CD instance and each connected cluster
Upgrade Argo CD instances on a new version in a controlled manner
Keep track of versions/security alerts, and easily upgrade

The unified control plane is the perfect tool for all organizations that need a management interface on top of all their Argo CD instances without all the hassle for manual upgrades.

Summary

In this blog post we have seen most Argo CD installation methods from the simplest one (just the manifests) to the most powerful one (the unified control plane). Depending on the size and complexity of your organization you should choose a management method that allows you to focus on the things that matter most – deploying applications, instead of handling the instances themselves

For more information on the hosted instance and the control plane sign-up with Codefresh. See also our best practices article and getting started guides. And of course, don’t forget to get GitOps with Argo CD Certified!

Deploying Microservices with GitOps

Hannah — Thu, 28 Apr 2022 20:50:52 +0000

Learn how to adopt GitOps for your microservices deployments.

This blog post will explain the highlights listed below and how it's possible to adopt GitOps and the benefits of deploying with Argo for your microservices.

What are microservices?
What are some of the challenges with microservices?
What is GitOps?
What is Argo CD?
How can GitOps and Argo CD help your deployments for your microservices?

Microservices

Microservices are a software architecture pattern for building scalable distributed applications. It's essentially a framework that helps break down monolithic architectures into small independent services that can communicate over common patterns like APIs, event-driven, or even messages. An organization might do this because monolithic architectures are tightly coupled and run as a single service. So, if a particular process of your application with a monolithic architecture experiences a spike in demand, this affects your entire architecture and must be scaled.

It's also important to note that any features or experiments made to this type of application become more complex and therefore can experience more issues and failures.

Breaking down a monolithic application with API services allows you to build and version each function of an application independently. Because of this, microservices enable an organization to run, modify, deploy and scale each service for an application much more quickly.

(include visual of monolithic architecture vs. microservice)

Challenges of Microservices

Microservices provide several benefits for organizations by enabling autonomy amongst development teams, allowing applications to scale, deploying those applications efficiently, and increasing flexibility and resilience for those applications. However, microservices include challenges that we will dive into and explain.

Those challenges are:

The exploding number of pipelines.

As the number of applications increases, the management of such pipelines becomes more complex. Often this can result in messy scripts and needing to leverage configuration management tools. With all of that comes maintenance, and operators spend a lot of time fixing pipelines. We have a blog post that dives deeper into this issue if you'd like to learn more.

Difficulty managing deployments (all at once, in sequence, and with dependencies).

While deploying a single microservice is more manageable than deploying a monolithic legacy application, there are still challenges. Suddenly, only a few pipelines for a monolithic application have turned into 20 or more pipelines with multiple microservices. These numbers will vary based on your organization, but it's no doubt that now managing multiple microservice repositories and pipelines is more complicated as more and more are created.

Difficult to monitor.

Now that there are multiple pipelines and multiple applications with an increase in deployments, those deployments and pipelines need to be monitored for any issues. Previously when using a monolithic architecture, these things could be monitored manually. However, the increased complexity of a distributed system can lead to chances of failure.

This is why it’s so important to ensure you’re monitoring for total failures, network failures, cascading failures caused by remote procedure calls between your microservices, and even ensuring that you’re honoring your SLAs (service level agreements). With a microservice architecture, we need to keep in mind that services can even be temporarily unavailable due to broken releases, configurations, and any changes due to human error. Clients of services should be architected in a resilient way so they do not suffer catastrophic failure when a service is down.

Difficult to debug (when things go wrong).

This topic itself could be a blog post. However, some of the critical challenges when debugging a microservice is fragmentation and ensuring that you create the same environment where the bug was identified. This means you need to ensure an identical setup to production, this may include specific cluster versions, serverless functions, the correct environment where the bug was reported, etc. In addition to the asynchronous call complications, the technicalities of running the microservice locally, and the cost it can take to execute all of these things in what you hope is a timely execution.

Even with the above example of the complexity that microservices can create when transitioning from a monolithic architecture to a microservice architecture, the benefits of moving to microservices can’t be understated. To gain the full benefits of microservices in scaling, reliability, and security we need to address these challenges head-on

So, what could help you with some of the challenges organizations face when managing, monitoring, deploying, and maintaining their microservices? Let me introduce you to GitOps, how you can apply it to your microservices, and how it might help!

GitOps

Before we dive into microservices and explain what they are and what sort of challenges they create for us. First, let's dive into what GitOps is so you can begin to percolate your thoughts as to WHY this might be helpful for your microservices as we explain them.

Perhaps you're familiar with Git and use it already for your source control and recognize how it enables a source of truth for your source code, but what if we expanded on that idea and applied it to our entire system? This is where GitOps comes into play.

GitOps is a paradigm that leverages a Git repository as the source of truth to manage continuous deployments and infrastructure management as code. This allows you to drive automation, bring repeatability, reduce downtime and eliminate human interventions in constant release cycles.

Now that we better understand GitOps let's explore a potential solution for deploying your microservices with GitOps that can be enabled by using Argo CD.

Argo CD

When determining the best deployment process and tools for GitOps, Argo CD is a great solution. Argo CD is a declarative CD (Continuous Delivery) tool that automatically synchronizes and deploys your applications whenever a change is made to your Git repository. This is why it’s one of the leading GitOps solutions.

It allows you to manage the lifecycle of your application and deployments while also providing version control, configuration management, and application definitions with a Kubernetes manifest. This allows you to organize your environments and complex infrastructure data with more ease.

Within this post, we will highlight ways that you can use Argo CD to deploy your microservices while applying GitOps to your workflow.

How GitOps and Argo can help

When trying to determine the best strategy to manage your Kubernetes applications, an increasingly popular approach is GitOps. Argo CD makes GitOps practical for you to use because Argo CD is a Kubernetes-native declarative continuous delivery tool that follows GitOps principles. Argo CD also enables you to accelerate the frequency of your application deployments and lifecycle management by maintaining your configurations and ensuring they’re in sync, applying your desired state directly from Git.

Essentially, Argo CD helps support your deployments while effectively providing several other benefits for your microservices which we’ll explain more below!

No more deployment pipelines -> Argo CD deploys them from Git

Previously we mentioned the explosion of pipelines when converting from a monolithic architecture to a microservice architecture. However, by leveraging Argo CD it can reduce the number of pipelines by simply eliminating them. This is because you can use Argo to deploy your microservice you rely on your Git repository instead of using a pipeline. When you add an Argo CD Application and an AppProject, your app's configuration settings can be defined using the Application Custom Resource (CRD) that your Kubernetes resource object represents.

Argo CD fully automates the deployment of an application by recognizing your configuration details from your manifest files and synchronizes any changes made to your system within your targeted namespace. Argo CD allows you to eliminate the use of a deployment pipeline.

Reduce files -> use an ApplicationSet

While microservices do make it easier to deploy more frequently, there’s also the increase in files for your configurations. Even when a service is deployed and managed using GitOps, this doesn’t reduce the number of files, such as your Kubernetes manifests, your services, secrets, and configmaps, all stored within your Git repository.

Luckily, Argo CD offers a controller known as an ApplicationSet to help us manage our deployments and files easier. The ApplicationSet allows you to create multiple Argo CD Applications and organize your microservices per Application. The controller also allows you to use a single manifest to target numerous clusters and a single manifest to deploy multiple applications, not just from one repository but multiple!

Always know their health -> Argo CD health

Whenever you make any change to your microservices or your system there’s a risk of a potential failure. For example, if you make a change and apply it to your manifest YAML file, Kubernetes does not wait until the resource reports back its status, nor do your automation tools typically report back, either. Instead, Kubernetes will apply the manifest and move on to the following manifest without waiting to know if the first one succeeded and was created or not. Something that can help alert you and provide observability for the state of your service is the Argo CD health status.

Argo CD offers a built-in health assessment that provides an overall Application health status for you. Argo CD checks the health for the following resources: services, ingress, deployment, replicaset, statefulset, daemonset, and a persistentvolumeclaim. You can also add custom health checks for custom resources or any persistent known issues affecting your Ingress or StatefulState. These custom health checks can be done two ways: within a ConfigMap or with a script and implemented with a YAML file. If you cannot find a health check for your resource, you can create one yourself for whatever use case you need it.

Above is an example of the Argo CD Application health status within the Argo CD Web UI. Since Git is our source of truth, this sort of observability and information about your service provides that source of truth for the actual state of your running system.

Deployment cadence -> Argo CD sync phases/waves

Typically when managing several microservices and making frequent deployments, it can be a challenge when needing to deploy a specific service in a particular order or, let's say, you want to deploy a service with a database, apart from deployment files. It would help if you had some solution for this. Whatever your use case may be, Argo CD provides a solution for this!

A Syncwave allows you to order how Argo CD applies the manifest files stored in your Git repository. By default, all manifests have a wave of zero, but you can set these by using an argocd.argoproj.io/sync-wave annotation.

For example, if you wanted to specify a wave, it would look something like this:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

When Argo CD starts a sync action, the manifest gets placed in the following order of their Phase.

When you apply the manifest in a Phase, you're using a resource hook to control the sync operation. Those operations are defined by the three primary phases known as the pre-sync, sync, and post-sync phases. To enable a sync, annotate the specific object manifest with argocd.argoproj.io/hook with the type of sync you want to use for that resource.

For example, if you wanted to use the PostSync hook, it would look something like this:

metadata:
  annotations:
    argocd.argoproj.io/hook: PostSync

Whenever Argo CD starts a sync, it orders the resources in the following precedence: the phase, the wave they are in, by kind (e.g., namespaces first and then other Kubernetes resources, followed by custom resources), and by name.

Dependencies -> See the graphical view of Argo CD

Now that you have containerized your microservices and wrapped your brain around the Kubernetes ecosystem. Suddenly an extra layer of complexity is added when trying to deploy all of those microservices and reference the never-ending Kubernetes documentation. This is where having some visualization for your deployments is helpful! Enabling some sort of topology for your application and infrastructure would allow you to better understand, monitor, and control your containerized microservice.

Argo CD does just that. The web UI provides a real-time view of application activity and dependencies and continuously monitors running applications when using Argo CD. Monitoring all deployed applications allows you to compare the current, live state against the desired target state (as specified within your Git repository). Any deployed application whose live state deviates from the target state is reported and allows you to view the differences while also providing facilities to automatically or manually sync the live state back to the desired target state of your system.

Image from Argo CD

The image above exemplifies an Argo CD Application with a topology view that shows all of the dependencies of that application.

Fast rollbacks/progressive delivery -> Argo Rollouts

When beginning to plan your deployment process for your new microservices, it's essential to determine which deployment strategy is best for you and your organization.

Potential strategies could include:

Rolling: Services that don't need to be available 24/7 can utilize a pattern of swapping in the new for old and essentially hitting the restart button. This allows minimal disruption should there be a service outage. The simplicity aspect is compelling, and the disruptions to end users are less so.

Progressive delivery (this includes blue-green deployments and canary): A blue-green deployment is simple, has excellent rollback, and has minimal downtime in best-case scenarios. Canary deployments tie together the best of all worlds for minimizing downtime, fast rollback, and minimal disruptions in the case of failed deployments.

However, while progressive delivery improves the reliability and stability of deployments, applying it within your complex microservice environments doesn't come easy. Kubernetes was built as a container orchestrator, not a deployment system, requiring manual work to execute progressive delivery. Kubernetes also has a default rolling update strategy that is simple to deploy yet provides little control over user traffic and rollback compared to the more advanced deployment methods like blue-green and canary.

This is where Argo Rollouts walks down the red carpet as an open source Kubernetes controller with a GitOps-based progressive delivery strategy.

Argo Rollouts introduces a new Kubernetes CRD for a rollout entity. It has the same functionality as a Kubernetes Deployment entity, but the Argo Rollouts will monitor for changes and can control the rollout. Users can define a specific rollout strategy configured with a manifest file. Argo Rollouts helps you simplify your release processes and provide full support of progressive delivery with automated deployment rollout and management in one!

Argo Rollout Image of UI Dashboard

This image above exemplifies a visualization of an individual Rollout using a Canary deployment strategy.

Conclusion

As you transition from a monolithic architecture to microservices, this process shares a common goal with GitOps: making it easier to deploy more frequently.

We have shared the benefits and possibilities of adopting GitOps with Argo for your microservices deployments within this blog post. To help you leverage these benefits and get started with your GitOps journey, make sure you understand what GitOps is and the fundamentals of this paradigm. To get started, you can check out our GitOps Fundamentals course for free, which allows you to learn about GitOps and get started with the basics of Argo, in tandem with receiving a certificate of completion to share with your employer and others!

Once you’ve completed this course, you should put your knowledge to the test and try Codefresh GitOps CD to help manage your Argo CD instances. This will provide you with a visualization of your services and any possible failures tied to a commit, a test, pipeline, etc. You can quickly pinpoint issues with access to a release timeline and functionality to execute rollbacks when needed.

Learn more about Codefresh GitOps CD and sign up here!

GitOps Benefits and Considerations

Tracy Holmes — Wed, 13 Apr 2022 21:41:53 +0000

GitOps. The term appears everywhere, but what are its benefits? And is it as difficult as it sounds? Well, GitOps is a pretty easy paradigm to integrate with your current processes. However, my saying it is “easy” doesn’t help you decide whether you want to adopt it or not. So, let’s talk about it.

This post will briefly discuss common software delivery challenges, DORA metrics and what they are, how GitOps can help with those metrics, and what to consider if you would like to adopt it.

Software Delivery Challenges

Most existing processes for infrastructure configuration management face challenges like failed deployments, poor infrastructure design, server outages, etc. Some common challenges are:

“Eventual Consistency” or lack of configuration consistency in your environment. For example, how do you know if “System A” and “System B” match? The actual configuration and a declared one can alter and change when using manual processes, even when using a centralized configuration.
No idea as to how or where an application is running. Implicit infrastructure, its state, and configuration. This sometimes entails a full-scale investigation to determine the “how” and the “where.”
Failed deployments relying on a disaster recovery strategy. With manual or semi-manual processes, disaster recovery requires a strict process that teams are disciplined to tackle, which isn’t always reliable—resulting in unplanned downtime that takes away from developers’ skills and productivity.
Missing documentation or unknown development history. It can be challenging to figure out how some applications were built initially and who managed them.
Relying on a system’s previous state for success. This can be unpredictable, which affects the stability and success of a project.

Or, to state these challenges in another way: inconsistent configurations can lead to failed deployments, leading to revenue loss. GitOps can help with all of these challenges. We are going to dive into GitOps more in a bit. But first, let’s talk about DORA metrics and why they matter.

DevOps and DORA metrics

DevOps is a collaboration paradigm, and sometimes it is mistaken for being too abstract or too generic. In an effort to quantify the benefits of adopting DevOps, Dora Research (acquired by Google in 2018) has introduced four key metrics that define specific goals for improving the software lifecycle in companies that are interested in DevOps.

These four key metrics and their organizational effects have been published as part of the “State of DevOps” reports since 2014. The reports include a survey (with more than 32,000 industry professionals in the last years) that categorize the organizations that responded. These categorizations or performance levels are:

Elite performers (companies that have fully adopted DevOps and have exceptional results in the four metrics)
High performers (companies with excellent results in the four metrics)
Medium performers (companies with some positive results)
Low performers (all other companies)

The researchers responsible for the survey apply these different categorizations by examining how teams develop, deliver, and operate software systems, and then segmenting respondents into four performance clusters: elite, high, medium, and low performers.

The 4 Key DevOps Metrics + 1

The four metrics of software delivery performance are:

Deployment Frequency (DF) – How frequently a team successfully releases to production, e.g., daily, weekly, monthly, yearly.
Lead Time for Changes (MLT) – The median amount of time for a commit to be deployed into production.
Time to Restore Service(s) (MTTR) – How long it takes an organization to recover from a failure in production. For a failure, the median amount of time between the deployment that caused the failure and the remediation. The remediation is measured by closing an associated bug/incident report.
Change Failure Rate (CR) – The number of failures per the number of deployments. For example, if there are four deployments in a day and one causes a failure, that is a 25% change failure rate.

2021 State of DevOps – Software Delivery Performance Metric

Since 2018, a fifth metric around operational performance has also been included: reliability.

Reliability – the degree to which a team can keep promises and assertions about the software they operate. Reliability was chosen so that availability, latency, performance, and scalability are more broadly represented.

Companies’ DevOps teams ranked as elite or high-performing in these five categories tend to have outstanding performance around market share and profits.

What is GitOps?

What exactly is GitOps? If you are not familiar with GitOps, head over to https://opengitops.dev, which is the official page of the GitOps working group. In short, GitOps is a set of standards and best practices that can be applied to application configuration and delivery, all the way to infrastructure management. GitOps is not a tool but works by relying on a single source of truth (typically Git) for declarative applications and infrastructure. The principles of GitOps are the following:

The system is described in a declarative manner.
The definition of the system is versioned and audited. This tends to be Git though it is not limited to it.
A software agent automatically pulls the Git state and matches the platform state.
The state is continuously reconciled. This means that any changes in Git should also be reflected in the system.

You will usually see GitOps paired with Kubernetes, though Kubernetes is not necessarily required. You can use GitOps to build pipelines, provision clusters, manage configurations, and much more.

Why GitOps is beneficial for DORA metrics

DORA metrics are great, especially when demonstrating critical areas for improvement to organizations. However, while organizations know about the metrics, it can be tough to have good performance numbers there. Adopting GitOps principles can help tackle these issues.

For example, because GitOps adds a structure of repeatability to your organization’s different environments, you can reduce MTTR and the change failure rate. This is because you now have a declarative reference architecture in place of your entire system. You also eliminate delivery as the last step in your Continuous Integration (CI) pipeline because of synchronization. That synchronization also means because you use Git for your systems’ versioning, you can now ensure that your deployments match your desired state. All of this combined means your MTTR is significantly reduced from days or hours to just minutes.

Also, you now have more frequent releases because you are using continuous deployment, which translates into increased speed and productivity. This increased speed and productivity translate into happier customers because of the potential for new features or functionality due to decreased turnaround time.

Know what is deployed where

Imagine having a magic wand that handles your deployments. GitOps can do that for you.Some CI tools follow an authoritative or ad-hoc approach and use things like custom kubectl scripts to deploy. The urge to change or edit something manually in environments is significant – it is perceived as quicker or easier. So, for example, you may have “cowboy deployments” or “cowboy engineering” where your DevOps team may be performing manual changes to the cluster. But when this happens, you have no idea what’s going on since those changes aren’t recorded anywhere, and the modifications can be pretty delicate.GitOps gets rid of those manual changes because whatever agent you’re using is in charge of the deployment. The deployment process or synchronization is a consolidation of whatever your current cluster state is to the desired state. The agent is also in charge of monitoring. Because of this, things are alike and in sync.

Faster deployments and rollbacks

There are times when you or a colleague may want to know, “What are the current deployment versions in environments A and B?” or “Is it possible to roll back to version XYZ?” When using a traditional CI/CD solution, these questions may be tough to answer because any number of ad-hoc patches, hotfixes, or other changes may have occurred since the initial deployment.

However, using GitOps, things are a bit simpler. Your Git repository will be able to tell you what deployments are in the cluster because your commit history can pretty much work as your cluster deployment history. The state of your cluster should look very similar to your latest git commit.

Rollbacks are also easier with GitOps. Things are more “hands-on” when working with traditional CI/CD. With conventional CI/CD, you have to research your deployment pipeline to find the correct version of what you need and then manually trigger it. With GitOps, this is not the case, as you only need to perform a reset in Git, and once things are synchronized, the cluster will reflect the necessary changes. Or, depending on what agent you are using, you could do something as simple as selecting the appropriate Git release and let the synchronization process handle the rest.

No configuration drift

Configuration drift is usually due to manual changes and updates across an organization’s environments. It is a problem that existed even with traditional Virtual Machines and has plagued production deployments long before Kubernetes appeared on the scene. The more environments you have and the longer your configuration drift is present, the more crucial things will become.

Remember when I mentioned monitoring earlier? Well, even though your agent’s sync process is crucial for performing the initial deployment of your application, one of the true strengths is the continuous monitoring of both states (cluster and Git) after the deployment takes place. The reconciler takes care of eliminating configuration drift by staying true to ONLY to the Git state. This uni-directional reconciliation is vital for solving configuration drift which is a widespread issue in organizations with large numbers of deployment targets.

If you would like to get an idea of how your team or organization’s software delivery performance compares to the rest of the industry, you can take the DORA “DevOps Quick Check” here.

Considerations for Adopting GitOps

There are two things to consider when deciding whether or not to embrace GitOps:

You need to have a CI in place already
You need a GitOps Agent

The first one is simple. If you already have a CI solution in place, keep using what you have. So, let's tackle the second.

If you need a GitOps agent, Argo CD is a great choice. Argo CD is a popular deployment solution for Kubernetes. When following a GitOps deployment pattern, Argo CD makes it easy to define a set of applications with their desired state in a repository and where the deployment should happen. After a deployment, Argo CD continuously monitors a Git repository with Kubernetes manifests and listens for commit events.

When a commit happens (usually one that updates the versions of the image artifacts), Argo CD starts a “synchronization” process responsible for bringing the cluster configuration to the same state as described in Git.

When the sync process is complete, we know that the application configuration is the same as the Git manifests.

The Argo CD deployment process is the embodiment of the core ideas behind GitOps that we discussed at the beginning:

All application configuration is stored in Git (usually in a separate repository than the source code)
Deployments are happening in a “pull” manner where the cluster is fetching manifests from Git (instead of traditional solutions where updates are “pushed” to the cluster)
A deployment is a uni-directional reconciliation process between the two states (what is described in Git versus what is deployed in the cluster)
Even though the sync process is vital for performing the initial deployment of the application, one of the true strengths of Argo CD is the continuous monitoring of both states (cluster and Git) after the deployment takes place. This continuous monitoring is essential for solving configuration drift – a common issue in organizations with a considerable number of deployment targets.

To clarify: your CI will stay the same. Nothing is changing except GitOps making things simpler. You no longer have to do certain things manually. If you were using Git? Continue to use Git. If you were using Kubernetes? Continue using Kubernetes. GitOps is less painful than you may imagine, and you are not starting over from scratch. Things are now streamlined.

You can learn how to get started with Argo CD here and read about Argo CD best practices here.

Conclusion

I mentioned earlier that GitOps was not just for Kubernetes. And that’s true! Kubernetes isn’t a requirement. GitOps can absolutely integrate with other deployment pipelines or infrastructure, including containers and VMs.

GitOps is also not just for applications. You can also use it with infrastructure toolings such as Terraform or Crossplane. If it is transactional, can be described declaratively, and Infrastructure-as-Code (IaC) tools are available, you can use GitOps in your environment.

Hopefully, now that you’ve learned more about GitOps you can find ways to introduce this paradigm to your workflow, whether it’s containerizing your application or adding your configuration to your repository. By doing so, you’ll be in good company: CapitalOne, Intuit, Tesla, Red Hat, Google, Codefresh, and many others are using Argo and GitOps to eliminate challenges in their environments. And if you’d like to learn more about how Codefresh has embraced GitOps or get started on your GitOps journey, you can try out the free Community Edition here.

Best Practices for Argo CD

Hannah — Mon, 04 Apr 2022 15:20:18 +0000

What are some best practices when using Argo CD?

Within this blog post, we’ll be highlighting some best practices tied to Argo CD, that allow you to leverage GitOps easily within your deployment workflow.
Below we will explain the following:

What is Argo CD
What are some best practices with Argo CD?

What is Argo CD

Argo CD is the most popular and fastest-growing GitOps tool for Kubernetes. When following a GitOps deployment pattern, Argo CD makes it easy to define a set of applications with their desired state in a repository and where they should be deployed. After a deployment, Argo CD constantly monitors the state and can even catch configuration drift.

Argo CD's core component is the Applications Controller that executes continuous monitoring of applications and then compares them to the live application state against your target state defined within your Git repository.

The Application controller retrieves the desired resource manifest from the Git repository and compares it to the live resource from your Kubernetes cluster.

This approach enables GitOps for your deployment workflow by leveraging Git as your source of truth. Now, let's further explore how we can leverage Argo CD best for our deployments!

The image above shows the primary Argo CD dashboard with a single Argo application successfully deployed.

The dashboard above provides a detailed view of the same Argo application deployed in the image above. However, it also provides an understanding of the status of the Kubernetes resources.

Now, let's explore how we can apply best practices to our Argo CD deployments!

Argo CD Best Practices

Argo CD has several best practices; however, we will review some of the most important ones we've gathered from the Argo community and prioritized below.

1. Separate your Git repositories

It's super important to separate your configurations and source code into different Git repositories. Separating your configs in their repository limits commit access to avoid something being pushed to production environments. For example, if you accidentally push manifest changes to the config repo, it can trigger an infinite loop of build jobs and Git commit triggers. Using an automated CI pipeline, you can also prevent pushing manifest changes to the same repo.

Separating your repositories is also more secure and restricts commit access, so someone does not accidentally misconfigure an application.

Argo CD does not connect to your source code repository. However, if you make a change to your code and you want to deploy the change to a specific environment, you can do the following:

Build a new application code version and then merge a pull request (PR) in the configuration repo to use the new application version you created.
Utilize automation for the update you made to your source code by using a pipeline in the source code repository.
Argo CD Image Updater allows you to update your container images of the Kubernetes workload managed by Argo CD.

These approaches allow you to maintain logs for any auditing process. You can quickly identify development activity and reference your Git history for easy traceability.

2. Create a directory structure to enable a multi-application system for your Argo CD deployments

Once you've separated your source code and configs into separate repositories, it's essential to set up a directory structure that applies GitOps for Argo CD deployments. You can leverage several approaches for your Git repository structure that best serves your organization's needs.

However, we'll highlight some tips to best structure your directory when using Argo CD.

Do: We suggest modeling your environments or clusters using different folders instead of branches in your configuration repository (e.g., prod, staging, testing, etc.).
Do: Make sure your cluster and environment configurations repositories are separated (i.e., separate your prod configuration in a different repository from staging).
Do: Utilize some sort of manifest management, such as a raw Kubernetes YAML file, Kustomize, or Helm for your environment definitions for your apps.
Do: Create an 'argocd' folder in your configuration repository for each cluster and create an Argo CD Application manifest for each app in the cluster's repository.
- By creating the separate 'argocd' folder, you can also implement role-based access control for different clusters if you wish with Git repository permissions.
Do: Leverage a multi-folder or a multi-repo structure instead of a multi-branch approach. You should NOT have permanent branches for your clusters or environments.
Don't: Never put any independent applications or applications managed by different teams in the same repository.

Implementing these strategies above for your directory structure provides several advantages. Those advantages include improved security, easy rollbacks, audit log, easier testing, and automating your manifest updates. It also allows you to create and delete applications dynamically. If you'd like to see some examples of these directory structures, look here.

3. Determine a promotion strategy

Now that you've established a directory structure, you may face a challenge regarding the best promotion practice between clusters. When deploying multiple applications with Argo CD, it's best to pick one promotion strategy that suits your directory structure and stick with it. Below we'll highlight how you can do this based on your own needs.

Group your applications

ApplicationSets

First and foremost, once you've established a way to manage your applications and deployments, and there are too many apps to keep track of - the ApplicationSet is your best solution. We should also note that the ApplicationSet was previously an external controller that you had to install on your own, and now it's integrated and makes things much easier! However, you need some extra support when using the ApplicationSet instead of the App of Apps pattern that is easy enough to use right away without any additional help or a learning curve.

Yet, when deploying to production, an ApplicationSet is the best choice. This is because an ApplicationSet is a Kubernetes controller/custom resource definition (CRD) that enables automation and allows flexibility when managing multiple applications across several clusters.

The ApplicationSet consists of two main components:

You are essentially allowing the ApplicationSet to act as a templating agent.

The generators used with the ApplicationSet are responsible for generating params, utilizing the template to consume the variables, and then applying them to the cluster as Argo CD applications. So, any changes made to an ApplicationSet template and its fields will automatically be applied to each application created.

This way, you can simultaneously deploy your Argo apps to multiple Kubernetes clusters.

App of Apps

When managing ten or fewer applications, it's best to use the App of Apps pattern.

App of Apps was the pioneer before ApplicationSets that allowed us to deploy multiple applications at once with ease.

You are leveraging the actual app itself, aka a "Root App," to contain the other applications instead of individual Kubernetes objects. This, in turn, allows you to manage a group of applications that you can deploy declaratively. Essentially this pattern supports the declaration of children apps in a recursive way.

Image from the Argo CD documentation

Another possible approach to promoting changes is the Argo CD Image Updater. It's still a relatively new project, but it provides a manual way to update your manifest versions.

However, we should note that this approach is not GitOps friendly. Meaning it goes against the declarative nature of GitOps, but perhaps its functionality is what you and your team needs.

Which to choose?

One does not "need" a promotion strategy when beginning to deploy with Argo CD. The strategies listed above are needed when trying to solve "too many apps." This problem dramatically depends on the number of Argo CD applications you are managing and whether or not you already have a deployment process in place or not.

The Argo community has developed the strategies listed above to help you manage all your Argo CD application deployments, depending on your scale, directory structure, and manifest types, amongst other factors.

However, it's ultimately up to you and your workload, which is best for you and your team.

4. Manage your secrets securely

No one solution for secret management will work for all organizations. However, some common approaches for how to best handle your secrets with Argo CD can help based on these two significant factors:

Encrypt your secrets directly in your Git repository:

Bitnami Sealed Secrets

Sealed secrets are one way to encrypt secrets created by anyone but can only be decrypted by the controller running within the target cluster.

SOPS

SOPS (Secret OperationS) is an open source solution that allows you to encrypt and decrypt an entire file or field for your Kubernetes secrets. This approach will enable you to store your secrets, and other Kubernetes manifests directly in your Git repository.

Externalizing your secrets from your Git repository:

Argo CD Vault plugin

This solution creates debate about whether it's GitOps or not. Nonetheless, it's a solution using Kubernetes auth in Vault. The Argo CD repo server authorizes Vault to use the service account token in the secret manifest, substituting the required value and creating a secret for you.

Cloud Provider Secrets

Cloud provider secrets are created by your cloud service providers (CSP) depending on the cloud service. It depends on what your CSP offers and if their secret management solution meets your needs. Ensure you evaluate which strategy secures your data best.

Firstly, storing your secrets in a Git repository isn't easy! Perhaps your organization has a different policy for secrets, or you already have a system in place, so in this case, you may want to externalize your secrets.

However, if you want to store your secrets in your Git repository, plaintext secrets cannot be stored in your repo, and you never want to risk storing sensitive data and exposing it!

But, Git can truly be the source of truth, including your secrets, and you won't have to risk getting fired when they are encrypted asymmetrically. You can use Argo CD to manage your deployment state from Git into your cluster, including your secrets, by ensuring processes are put in place, and the tools above are used safely.

5. Confirm how your team accesses Argo CD

Essentially, we cannot say what the best practice for you and your team is to access Argo CD. However, we can advise you and your team to choose which approach works best for your organization and set a standard for your team.

Developers require access to Argo CD, so you need to set security measures and role-based access control (RBAC).

This approach allows your developers to access the Argo CD UI and ensure that each team has the appropriate access to their applications and their applications only.

Argo CD provides an RBAC configuration that includes two pre-defined roles:

Read-only
Admin

You can also implement a permissions definition for your applications and other resource types with Argo CD. You can then sync these roles and permissions together to create a policy to define access to your system.

No one within your organization requires access to Argo CD.

Some organizations do not want anyone to access the Argo CD UI or the API server. Perhaps the organization would like to use the automation but does not want to leverage the UI component. In this case, you can use a variant of Argo CD, also known as Argo CD Core. This allows developers to make a commit and ignore the UI aspects and reduce complexity. However, we should note that you lose multi-tenancy benefits by installing Argo CD Core.

So, whichever approach you choose above - as long as it benefits you and your team, you're making the right choice.

6. Increase automation for your system with the other Argo projects

If you're using Argo CD for Continuous Delivery (CD), you want to elevate the rest of your workflow to automate everything in Kubernetes to reduce downtime. Then, you should check out the other Argo projects and learn more about how they can help.

We don't encourage you to use any specific projects to leverage best practices, but instead, we encourage you to use those that will help you and your system best.

In the meantime, we can explore how we could use Argo Events, Workflows, and Rollouts to enhance your system if you choose to.

Progressive Delivery

Progressive delivery is a set of practices that roll out new features gradually instead of all at once.

Argo Rollouts

Rollouts provide advanced deployment capabilities and rolling updates for progressive delivery approaches you might already know, such as blue-green, canary, etc.

Advanced Deployment with CI

Argo Workflows and Events require you to have an existing Continuous Integration (CI) process to leverage these projects.

Argo Workflows

Workflows allow you to build and orchestrate parallel jobs and utilize a pipeline on Kubernetes.

Argo Events

Events is an event-driven workflow automation framework that is used with Kubernetes.

By utilizing these other Argo projects, you can easily manage your clusters, run workflows, and implement GitOps for your Kubernetes system! However, not all organizations are ready to implement automation for everything in Kubernetes. Combining all these tools requires time and understanding of how your system works before identifying which tool will best help your team's process.

Conclusion

Argo CD and the other Argo projects certainly make life easier by allowing you to automate each step for a production release, migration, and the operation workflows for your system!

By leveraging these best practices above, you're on the right track to empower your teams and apply GitOps for your infrastructure so you can deploy to prod more frequently.

I'd like to thank those who are part of the Argo community who participated and provided feedback for this blog. If you have any questions or think there are better ways to work with Argo CD, please reach out and let me know!

How to Model Your Gitops Environments and Promote Releases between Them

Kostis Kapelonis — Wed, 23 Mar 2022 11:49:56 +0000

Two of the most important questions that people ask themselves on day 2 after adopting GitOps are:

How should I represent different environments on Git?
How should I handle promoting releases between environments?

In the previous article of the series, I focused on what NOT to do and explained why using Git branches for different environments is a bad idea. I also hinted that the “environment-per-folder” approach is a better idea. This article has proved hugely popular and several people wanted to see all the details about the suggested structure for environments when folders are used.

In this article, I am going to explain how to model your GitOps environments using different folders on the same Git branch, and as an added bonus, how to handle environment promotion (both simple and complex) with simple file copy operations.

Hopefully this article will help with the endless stream of questions and discussions on this hot topic.

Learn your application first

Before creating your folder structure you need to do some research first and understand the “settings” of your application. Even though several people talk about application configuration in a generic manner, in reality not all configuration settings are equally important.

In the context of a Kubernetes application, we have the following categories of “environment configuration”:

The application version in the form of the container tag used. This is probably the most important setting in a Kubernetes manifest (as far as environment promotions are concerned). Depending on your use case, you might get away with simply changing the version of the container image. However, several times a new change in the source code also requires a new change in the deployment environment
Kubernetes specific settings for your application. This includes the replicas of the application and other Kubernetes related information such as resource limits, health checks, persistent volumes, affinity rules, etc.
Mostly static business settings. This is the set of settings that are unrelated to Kubernetes but have to do with the business of your application. It might be external URLs, internal queue sizes, UI defaults, authentication profiles, etc. By “mostly static,” I mean settings that are defined once for each environment and then never change afterwards. For example, you always want your production environment to use production.paypal.com and your non-production environments to use staging.paypal.com. This is a setting that you never want to promote between environments
Non-static business settings. This is the same thing as the previous point, but it includes settings that you DO want to promote between environments. This could be a global VAT setting, your recommendation engine parameters, the available bitrate encodings, and any other setting that is specific to your business.

It is imperative that you understand what all the different settings are and, more importantly, which of them belong to category 4 as these are the ones that you also want to promote along with your application version.

This way you can cover all possible promotion scenarios:

Your application moves from version 1.34 to 1.35 in QA. This is a simple source code change. Therefore you only need to change the container image property in your QA environment.
Your application moves from version 3.23 to 3.24 in Staging. This is not a simple source code change. You need to update the container image property and also bring the new setting “recommender.batch_size” from QA to staging.

I see too many teams that don’t understand the distinction between different configuration parameters and have a single configuration file (or mechanism) with values from different areas (i.e. both runtime and application business settings).

Once you have the list of your settings and which area they belong to, you are ready to create your environment structure and optimize the file copy operations for the settings that change a lot and need to be moved between environments.

Example with 5 GitOps environments and variations between them

Let’s see an actual example. I thought about doing the classic QA/Staging/Production trilogy, but this is rather boring so let’s dive into a more realistic example.

We are going to model the environment situation mentioned in the first article of the series. The company that we will examine has 5 distinct environments:

Load Testing
Integration Testing
QA
Staging
Production

Then let’s assume that the last 2 environments are also deployed to EU, US, and Asia while the first 2 also have GPU and Non-GPU variations. This means that the company has a total of 11 environments.

You can find the suggested folder structure at https://github.com/kostis-codefresh/gitops-environment-promotion. All environments are different folders in the same branch. There are NO branches for the different environments. If you want to know what is deployed in an environment, you simply look at envs/ in the main branch of the repo.

Before we explain the structure, here are some disclaimers:

Disclaimer 1: Writing this article took a long time because I wasn’t sure if I should cover Kustomize or Helm or plain manifests. I chose Kustomize as it makes things a bit easier (and I also mention Helm at the end of the article). Note however that the Kustomize templates in the example repo are simply for illustration purposes. The present article is NOT a Kustomize tutorial. In a real application, you might have Configmap generators, custom patches and adopt a completely different “component” structure than the one I am showing here. If you are not familiar with Kustomize, spend some time understanding its capabilities first and then come back to this article.

Disclaimer 2: The application I use for the promotions is completely dummy, and its configuration misses several best practices mainly for brevity and simplicity reasons. For example, some deployments are missing health checks, and all of them are missing resource limits. Again, this article is NOT about how to create Kubernetes deployments. You should already know how proper deployment manifests look. If you want to learn more about production-grade best practices, then see my other article at https://codefresh.io/kubernetes-tutorial/kubernetes-antipatterns-1/

With the disclaimers out of the way, here is the repository structure:

The base directory holds configuration which is common to all environments. It is not expected to change often. If you want to do changes to multiple environments at the same time, it is best to use the “variants” folder.

The variants folder (a.k.a mixins, a.k.a. components) holds common characteristics between environments. It is up to you to define what exactly you think is “common” between your environments after researching your application as discussed in the previous section.

In the example application, we have variants for all prod and non-prod environments and also the regions. Here is an example of the prod variant that applies to ALL production environments.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: simple-deployment
spec:
  template:
    spec:
      containers:
      - name: webserver-simple
        env:
        - name: ENV_TYPE
          value: "production"
        - name: PAYPAL_URL
          value: "production.paypal.com"   
        - name: DB_USER
          value: "prod_username"
        - name: DB_PASSWORD
          value: "prod_password"                     
        livenessProbe:
          httpGet:
            path: /health
            port: 8080

In the example above, we make sure that all production environments are using the production DB credentials, the production payment gateway, and a liveness probe (this is a contrived example, please see disclaimer 2 at the start of this section). These settings belong to the set of configuration that we don’t expect to promote between environments, but we assume that they will be static across the application lifecycle.

With the base and variants ready, we can now define every final environment with a combination of those properties.

Here is an example of the staging ASIA environment:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: staging
namePrefix: staging-asia-

resources:
- ../../base

components:
  - ../../variants/non-prod
  - ../../variants/asia

patchesStrategicMerge:
- deployment.yml
- version.yml
- replicas.yml
- settings.yml

First we define some common properties. We inherit all configuration from base, from non-prod environments, and for all environments in Asia.

The key point here is the patches that we apply. The version.yml and replicas.yml are self-explanatory. They only define the image and replicas on their own and nothing else.

The version.yml file (which is the most important thing to promote between environments) defines only the image of the application and nothing else.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: simple-deployment
spec:
  template:
    spec:
      containers:
      - name: webserver-simple
        image: docker.io/kostiscodefresh/simple-env-app:2.0

The associated settings for each release that we DO expect to promote between environments are also defined in settings.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: simple-deployment
spec:
  template:
    spec:
      containers:
      - name: webserver-simple
        env:
        - name: UI_THEME
          value: "dark"
        - name: CACHE_SIZE
          value: "1024kb"
        - name: PAGE_LIMIT
          value: "25"
        - name: SORTING
          value: "ascending"    
        - name: N_BUCKETS
          value: "42"

Feel free to look at the whole repository to understand the way all kustomizations are formed.

Performing the initial deployment via GitOps

To deploy an application to its associated environment, just point your GitOps controller to the respective “env” folder and kustomize will create the complete hierarchy of settings and values.

Here is the example application as it runs in Staging/Asia.

You can also use Kustomize on the command line to preview what is going to be deployed for each environment. Examples:

kustomize build envs/staging-asia
kustomize build envs/qa
kustomize build envs/integration-gpu

You can of course pipe the output to kubectl to deploy each environment, but in the context of GitOps, you should always let your GitOps controller deploy your environments and avoid manual kubectl operations.

Comparing the configuration of two environments

A very common need for a software team is to understand what is different between two environments. I have seen several teams who have the misconception that only with branches you can easily find differences between environments.

This could not be further from the truth. You can easily use mature file-diffing utilities to find what is different between environments just by comparing files and folders.

The simplest way is to diff only the settings that are critical to the app.

vimdiff envs/integration-gpu/settings.yml envs/integration-non-gpu/settings.yml

And with the help of kustomize, you can compare any number of whole environments for the full picture:

kustomize build envs/qa/> /tmp/qa.yml
kustomize build envs/staging-us/ > /tmp/staging-us.yml
kustomize build envs/prod-us/ > /tmp/prod-us.yml
vimdiff /tmp/staging-us.yml /tmp/qa.yml /tmp/prod-us.yml

I personally don’t see any disadvantage between this method and performing “git diff” between environment branches.

How to perform promotions between GitOps environments

Now that the file structure is clear, we can finally answer the age-old question “how do I promote releases with GitOps”?

Let’s see some promotion scenarios. If you have been paying attention to the file structure, you should already understand how all promotions resolve to simple file copy operations.

Scenario: Promote application version from QA to staging environment in the US:

cp envs/qa/version.yml envs/staging-us/version.yml
commit/push changes

Scenario: Promote application version from integration testing (GPU) to load testing (GPU) and then to QA. This is a 2 step process

cp envs/integration-gpu/version.yml envs/load-gpu/version.yml
commit/push changes
cp envs/load-gpu/version.yml envs/qa/version.yml
commit/push changes

Scenario: Promote an application from prod-eu to prod-us along with the extra configuration. Here we also copy our setting file(s).

cp envs/prod-eu/version.yml envs/prod-us/version.yml
cp envs/prod-eu/settings.yml envs/prod-us/settings.yml
commit/push changes

Scenario: Make sure that QA has the same replica count as staging-asia

cp envs/staging-asia/replicas.yml envs/qa/replicas.yml
commit/push changes

Scenario: Backport all settings from qa to integration testing (non-gpu variant)

cp envs/qa/settings.yml envs/integration-non-gpu/settings.yml
commit/push changes

Scenario: Make a global change to all non-prod environments at once (but see also next section for some discussion on this operation)

Make your change in variants/non-prod/non-prod.yml
commit/push changes

Scenario: Add a new configuration file to all US environments (both production and staging).

Add the new manifest in the variants/us folder
Modify the variants/us/kustomization.yml file to include the new manifest
commit/push changes

In general, all promotions are just copy operations. Unlike the environment-per-branch approach, you are now free to promote anything from any environment to any other environment without any fear of taking the wrong changes. Especially when it comes to back-porting configuration, environment-per-folder really shines as you can simply move configuration both “upwards” and “backwards” even between unrelated environments.

Note that I am using cp operations just for illustration purposes. In a real application, this operation would be performed automatically by your CI system or other orchestration tool. And depending on the environment, you might want to create a Pull Request first instead of directly editing the folder in the main branch.

Making changes to multiple environments at once

Several people have asked in the comments of the first article about the use-case of changing multiple environments at once and how to achieve and/or prevent this scenario.

First of all, we need to define what exactly we mean by “multiple” environments. We can assume the following 2 cases.

Changing multiple environments at once that are on the same “level.” As an example, you want to make a change that affects prod-us, prod-eu and prod-asia at the same time
Changing multiple environments at once that are NOT on the same level. As an example, you want to make a change to “integration” and “staging-eu” at the same time

The first case is a valid scenario, and we will cover this below. However, I consider the second scenario an anti-pattern. The whole point of having different environments is to be able to release things in a gradual way and promote a change from one environment to the next. So if you find yourself deploying the same change in environments of different importance, ask yourself if this is really needed and why.

For the valid scenario of deploying a single change to multiple “similar” environments, there are two strategies:

If you are absolutely certain that the change is “safe” and you want it to reach all environments at once, you can make that change in the appropriate variant (or respective folders). For example, if you commit/push a change in the variants/non-prod folder then all non-production environments will get this change at the same time. I am personally against this approach because several changes look “safe” in theory but can be problematic in practice
The preferable approach is to apply the change to each individual folder and then move it to the “parent” variant when it is live on all environments.

Let’s take an example. We want to make a change that affects all EU environments (e.g. a GDPR feature). The naive way would be to commit/push the configuration change directly to variants/eu folder. This would indeed affect all EU environments (prod-eu and staging-eu). However this is a bit risky, because if the deployment fails, you have just brought down a production environment.

The suggested approach is the following:

Make the change to envs/staging-eu first
Then make the same change to envs/prod-eu
Finally, delete the change from both environments and add it in variants/eu (in a single commit/push action).

You might recognize this pattern from gradual database refactorings. The final commit is “transitional” in the sense that it doesn’t really affect any environments in any way. Kustomize will create the exact same definition in both cases. Your GitOps controller shouldn’t find any differences at all.

The advantages of this approach are of course the easy way to rollback/revert the change as you move it through environments. The disadvantage is the increased effort (and commits) you need to promote the change to all environments, but I believe that the effort outweighs the risks.

If you adopt this approach, it means that you never apply new changes to the base folder directly. If you want a change to happen to all environments, you first apply the change to individual environments and/or variants and then backport it to the base folder while simultaneously removing it from all downstream folders.

The advantages of the “environment-per-folder” approach

Now that we have analyzed all the inner workings of the “environment-per-folder” approach, it is time to explain why it is better than the “environment-per-branch” approach. If you have been paying attention to the previous sections, you should already understand how the “environment-per-folder” approach directly avoids all the problems analyzed in the previous article.

The most glaring issues with environment branches is the order of commits and the danger of bringing unwanted changes when you merge from one environment to another. With the folder approach, this problem is completely eliminated:

The order of commits on the repo is now irrelevant. When you copy a file from one folder to the next, you don’t care about its commit history, just its content
By only copying files around, you only take exactly what you need and nothing else. When you copy envs/qa/version.yml to envs/staging-asia/version.yml you can be certain that you only promote the container image and nothing else. If somebody else has changed the replicas in the QA environment in the meantime, it doesn’t affect your promotion action.
You don’t need to use git cherry-picks or any other advanced git method to promote releases. You only copy files around and have access to the mature ecosystem of utilities for file processing.
You are free to take any change from any environment to either an upstream or downstream environment without any constraints about the correct “order” of environments. If for example you want to backport your settings from production US to staging US, you can do a simple copy operation of envs/prod-us/settings.yml to envs/staging-us/settings.yml without the fear that you might take inadvertently unrelated hotfixes that were supposed to be only in production.
You can easily use file diff operations to understand what is different between environments in all directions (both from source and target environments and vice versa)

I consider these advantages very important for any non-trivial application, and I bet that several “failed deployments” in big organizations could be directly or indirectly attributed to the problematic environment-per-branch model.

The second problem mentioned in the previous article was the presence of configuration drift when you merge a branch to the next environment. The reason for this is that when you do a “git merge,” git only notifies you about the changes it will bring, and it doesn’t say anything about what changes are already in the target branch.

Again this problem is completely eliminated with folders. As we said already, file diff operations have no concept of “direction.” You can copy any setting from any environment either upwards or downwards, and if you do a diff operation on the files, you will see all changes between environments regardless of their upstream/downstream position.

The last point about environment branches was the linear complexity of branches as the number of environments grows. With 5 environments, you need to juggle changes between 5 branches, and with 20 environments, you need to have 20 branches. Moving a release correctly between a large number of branches is a cumbersome process, and in the case of production environments, it is a recipe for disaster.

With the folder approach, the number of branches is not only static but it is exactly 1. If you have 5 environments you manage them all with your “main” branch, and if you need more environments, you only add extra folders. If you have 20 environments, you still need a single Git branch. Getting a centralized view on what is deployed where is trivial when you have a single branch.

Using Helm with GitOps environments

If you don’t use Kustomize but prefer Helm instead, it is also possible to create a hierarchy of folders with “common” stuff for all environments, specific features/mixins/components, and final folders specific to each environment.

Here is how the folder structure would look like

chart/
  [...chart files here..]
common/
  values-common.yml
variants/
  prod/
     values-prod.yml
  non-prod/
    Values-non-prod.yml
  [...other variants…]
 envs/
     prod-eu/
           values-env-default.yaml
           values-replicas.yaml
           values-version.yaml
           values-settings.yaml
   [..other environments…]

Again you need to spend some time to examine your application properties and decide how to split them into different value files for optimal promotion speed.

Other than this, most of the processes are the same when it comes to environment promotion.

Scenario: Promote application version from QA to staging environment in the US:

cp envs/qa/values-version.yml envs/staging-us/values-version.yml
commit/push changes

Scenario: Promote application version from integration testing (GPU) to load testing (GPU) and then to QA. This is a 2 step process

cp envs/integration-gpu/values-version.yml envs/load-gpu/values-version.yml
commit/push changes
cp envs/load-gpu/values-version.yml envs/qa/values-version.yml
commit/push changes

Scenario: Promote an application from prod-eu to prod-us along with the extra configuration. Here we also copy our setting file(s).

cp envs/prod-eu/values-version.yml envs/prod-us/values-version.yml
cp envs/prod-eu/values-settings.yml envs/prod-us/values-settings.yml
commit/push changes

It is also critical to understand how Helm (or your GitOps agent which handles Helm) works with multiple value files and the order in which they override each other.

If you want to preview one of your environments, instead of “kustomize build” you can use the following command

helm template chart/ --values common/values-common.yaml --values variants/prod/values-prod.yaml –values envs/prod-eu/values-env-default.yml –values envs/prod-eu/values-replicas.yml –values envs/prod-eu/values-version.yml –values envs/prod-eu/values-settings.yml

You can see that Helm is a bit more cumbersome than Kustomize, if you have a large number of variants or files in each environment folder.

The “environment-per-git-repo” approach

When I talk with big organizations about the folder approach, one of the first objections I see is that people (especially security teams) don’t like to see a single branch in a single Git repository that contains both prod and non-prod environments.

This is an understandable objection and arguably can be the single weak point of the folder approach against the “environment-per-branch” paradigm. After all, it is much easier to secure individual branches in a Git repository instead of folders in a single branch.

This problem can be easily solved with automation, validation checks, or even manual approvals if you think it is critical for your organization. I want to stress again that I only use “cp” in the file operations for promoting releases just for illustration purposes. It doesn't mean that an actual human should run cp manually in an interactive terminal when a promotion happens.

Ideally you should have an automated system that copies files around and commits/pushes them. This can be your Continuous Integration (CI) system or other platform that deals with your software lifecycle. And if you still have humans that make the changes themselves, they should never commit to “main” directly. They should open a Pull Request instead. Then you should have a proper workflow that checks that Pull Request before merging.

I realize however that some organizations are particularly sensitive to security issues and they prefer a bulletproof approach when it comes to Git protection. For these organizations, you can employ 2 Git repositories. One has the base configuration, all prod variants, and all prod environments (and everything else related to production) while the second Git repository has all non-production stuff.

This approach makes promotions a bit harder, as now you need to checkout 2 git repositories before doing any promotion. On the other hand, it allows your security team to place extra security constraints to the “production” Git repository, and you still have a static number of Git repositories (exactly 2) regardless of the amount of environments you deploy to.

I personally consider this approach an overkill that, at least to me, shows a lack of trust against developers and operators. The discussion on whether or not people should have direct access to production environments is a complex one and probably deserves a blog post on its own.

Embrace folders and forget branches

We hope that with this blog post we addressed all the questions that arose from the “don’t use branches for environments” article and you now have a good understanding about the benefits of the folder approach and why you should use it.

If you have other interesting use cases or have extra questions on the subject of organizing your GitOps environments, please ask in the comments section.

Happy GitOps deployments!

Getting Started with GitOps and Argo CD

Tracy Holmes — Thu, 24 Feb 2022 01:15:42 +0000

Today we are going to explore getting started using Argo CD. This post is going to assume you know a bit about containers, and that you already have an empty cluster in place (or know how to create one). If any of this is unfamiliar, head over to Understanding the Basics to get a bit of practice. Before we get started, let’s talk about GitOps.

What is GitOps?

If you are not familiar with GitOps, head over to https://opengitops.dev which is the official page of the GitOps working group. In short, GitOps is a paradigm that incorporates best practices applied to the application development workflow all the way to the operating infrastructure of a system. The principles of GitOps are the following:

The system is described in a declarative manner.
The definition of the system is versioned and audited. This tends to be Git.
A software agent automatically pulls the Git state and matches the platform state.
The state is continuously reconciled. This means that any changes happening in Git should also be reflected in the system.

A GitOps tool that follows this approach of a Git-based workflow is Argo CD. It’s a continuous delivery tool for Kubernetes that is essentially a GitOps controller that does two-way synchronization. Argo continuously monitors running applications, compares the live state against the desired state in Git, and applies it to the cluster. Please note this monitoring is for both directions: Git > cluster AND cluster > Git!

GitOps relies on Git as the single source of truth for declarative configuration and active reconciliation. What does this mean?

This means Argo CD implements active reconciliation by automatically monitoring your cluster and detecting any manual changes that are not in the Git state. Adopting the GitOps methodology provides transparency between the application configuration deployed in a cluster and the one residing in Git.

Additionally, some benefits of GitOps are deploying faster and more often, easier and quicker error handling and recovery, and self-documenting deployments. And most importantly? You also get the complete elimination of configuration drift. These benefits make it easier to handle the applications and allow teams to deliver quality software faster.

Prerequisites

For this exercise, you can absolutely use a local cluster such as Docker Desktop, minikube, k3s, etc. And if you have access to a cloud cluster? That’s even better.

However, if you don’t have access to those things, let's make sure we have kubectl installed. kubectl is a Kubernetes command-line tool that allows you to run commands against Kubernetes clusters.

We also need a kubeconfig file. If you already have an empty cluster, then that part of things has most likely already been done for you! In case you want to check, the default location for the config file is ~/.kube/config

Install Argo CD

Let’s start off by installing Argo CD. To do this, we’ll first create a new namespace, argocd, where Argo CD services and application resources will live.

kubectl create namespace argocd

Next, let’s apply our resource configuration file on the argocd namespace we just created.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Once your installation completes, you can use the watch command to check the status of your Kubernetes pods and make sure they are ready.

watch kubectl get pods -n argocd

Since we are using a stock Argo CD installation, you should see five pods with a Running status.

NAME                                  READY   STATUS    RESTARTS   AGE
argocd-redis-d486999b7-cd275          1/1     Running   0          8m57s
argocd-dex-server-65bf5f4fc7-p6kv8    1/1     Running   0          8m57s
argocd-repo-server-8465d84869-z4mqb   1/1     Running   0          8m57s
argocd-application-controller-0       1/1     Running   0          8m56s
argocd-server-87b47d787-pbnw8         1/1     Running   0          8m57s

Once things are up and running, you can use Ctrl+C to exit the watch interface. This is only one of the possible ways to install Argo CD! You are also able to use a Helm chart or a smart installer like Argo Autopilot.

Access The Argo CD API Server

By default, the Argo CD API server is not exposed with an external IP. For the purposes of this post, we are going to utilize kubectl port-forwarding to connect to the API server without actually exposing the service. We do that using

kubectl port-forward svc/argocd-server -n argocd 8080:443

This will make things easier for us as we can now access the API server using localhost:8080. Note: this method is only for local development or demonstration purposes.

There are two other techniques you can use to access the API server: Service Type Load Balancer and Ingress. If you’d like to learn more about those methods, definitely check out the Getting Started guide for more information.

Download Argo CD CLI

Now that we’ve gotten Argo CD downloaded, installed, and configured for access, it is time to login. There are two ways we can get this done: via the UI (or web interface) and the CLI. The UI and the CLI are mostly similar in capabilities. The CLI is great for changing settings and working with your Argo CD instance. Also, while you can deploy an application using the web interface, it’s typically quicker to deploy via the command line. I tend to use both depending on what I need. We will use both throughout this post.

I also work across different operating systems, so Homebrew is usually my goto installation method as it is available for Linux, Mac, and WSL. To install Argo CD using Homebrew, you will use the following:

brew install argocd

and check that it has installed correctly using

argocd version

You can also download the latest Argo CD version from https://github.com/argoproj/argo-cd/releases/latest. If you run into any issues, more detailed installation instructions can be found via the CLI installation documentation.

Login Using The CLI

We are going to login using an admin account. The initial password for this account is auto-generated and stored in your namespace as clear text in the field password and in a secret named argocd-initial-admin-secret. Let’s retrieve this password using kubectl:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo

I’ve added && echo to the end of that command to make it easier for us to copy & paste the generated password. Make sure to save it, as we will need it in the next steps.

Now that we have that password, we can login! So, let’s do that using

argocd login <ARGOCD_SERVER>

You will substitute Argo CD’s IP or hostname where you see <ARGOCD_SERVER> in that command. Remember when we set up port forwarding for easy access earlier in this post? Now it will come in handy!

argocd login localhost:8080

This will prompt you for a username and password. The default username is admin, and the password is the one we exposed up above. You will not see the password while entering it.

We also want to change the password while we’re here as the generated one was in cleartext, and that’s not safe. The following command will ask you for your current password, the new password you’d like to use, and lastly, to confirm the new password.

argocd account update-password

Once you do so, make sure to place your new password somewhere safe, as you will need it.

Create An Application From A Git Repository

In order for us to create an application to demonstrate how Argo CD works, we are going to use an example repository containing a guestbook application. If you would like to check it out or follow along, it is available at https://github.com/argoproj/argocd-example-apps.git.

Remember how I said earlier there are two ways we can get things done? You can definitely use one or the other (or both!) according to personal preference. I am a big fan of using the UI when first learning something in order to get a better idea of things I can do. So let’s start there to create our app.

Creating Apps Via UI

Open a browser to the Argo CD external UI by visiting localhost:8080, and login using the admin credentials we set up earlier. Your login page should look similar to this:

After logging in, click the “+ New App” button as shown below:

Give your app the name guestbook, use the project default, and leave the sync policy as Manual:

Connect the https://github.com/argoproj/argocd-example-apps.git repo to Argo CD by setting repository URL to the GitHub repo URL, leave revision as HEAD, and set the path to guestbook:

For Destination, set cluster to in-cluster and namespace to default:

We use in-cluster because when deploying internally (i.e. to the same cluster that Argo CD is running in), the https://kubernetes.default.svc hostname should be used as the application's Kubernetes API server address.

After filling out the information above, click Create at the top of the UI to create the guestbook application:

Creating Apps Via CLI

To create that same app using the CLI, you would use

argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default

When I do use the CLI, I still like to double-check myself using the UI occasionally. Do whichever is comfortable for you!

7. Sync (Deploy) The Application

Syncing via UI

Once the guestbook application is created, you can now view its status:

As you can see, the application status is OutOfSync! OutOfSync means the git state is different from the cluster state. You can see this information in multiple ways while in the UI. When the state is not in sync, you can usually tell as it is in a helpful yellow color. You can also filter by status on the left side of the dashboard. I highly recommend poking around the dashboard to see which configuration works for you!

That said, the application status is initially in OutOfSync state since the application has yet to be deployed. Also, no Kubernetes resources have been created. To sync (deploy) the application, we’ll use the “Sync” button.

This will bring up a second set of options. Select “Synchronize”.

Synchronizing is how we deploy the application. When we “Sync” or “Synchronize”, it means we are updating the cluster state to match the Git state. This is one of the most important tenets of GitOps.

Once things are complete and you return to the dashboard, you will see the guestbook app is now running. Select the app and see what information you can find!

If you click on the application name, you will see a hierarchical view of all the Kubernetes resources that take part in the application. You will also see additional details related to the health and synchronization status.

Syncing via CLI

Of course, everything we have done via the UI, can be done via CLI. Since we created the guestbook application, you can now check the status of your application using argocd app get:

argocd app get guestbook

Name:               guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:8080/applications/guestbook
Repo:               https://github.com/howmybrainworks/argocd-example-apps.git
Target:
Path:               guestbook
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from  (13b08f1)
Health Status:      Missing

GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH   HOOK  MESSAGE
       Service     default    guestbook-ui  OutOfSync  Missing
apps   Deployment  default    guestbook-ui  OutOfSync  Missing

If you look at the “Sync Status” you will see the application status is initially in OutOfSync state since the application has yet to be deployed, and no Kubernetes resources have been created. To sync (deploy) the application, run:

argocd app sync guestbook

Modifying and redeploying the Application

Here is where I feel Argo CD really shines. Now that we have a healthy system, let's initiate a change. Let’s scale the deployment by modifying the replicas in the deployment.yaml file.

In the deployment.yaml file, increase the number of replicas from 1 to 2. Since we are following GitOps, we want our modification to be declarative. So, we are going to commit and submit this change via Git. This will show things OutOfSync as well as identify configuration drift.

Resyncing (redeploying) via UI

If you go back to the UI, you should now see that we are now once again OutOfSync!

I sincerely love this feature, because not only does Argo CD let us know the application is no longer in sync, it also gives you a nice “App Diff” view to give more detail about what was changed! Click on the application name again, and you will see this option at the top of the page.

Now, the default “Diff” view can potentially contain an enormous amount of information, so I like to select the “Compact Diff” option in order to immediately see any changes. As you can see, we increased our replicas from 1 > 2.

Follow the same steps to deploy an application by using the “Sync” button, and we should be back to a healthy app.

Resyncing (redeploying) via CLI

Let's initiate another change. Let’s scale the deployment by modifying the number of replicas using kubectl.

Using kubectl, decrease the number of replicas from 2 to 1. You’ll do this using

kubectl scale --current-replicas=2 --replicas=1 deployment/guestbook-ui

Follow the same steps as above to retrieve our application status using

argocd app get guestbook

Name:               guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:8080/applications/guestbook
Repo:               https://github.com/howmybrainworks/argocd-example-apps.git
Target:
Path:               guestbook
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from  (13b08f1)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH   HOOK  MESSAGE
       Service     default    guestbook-ui  Synced     Healthy        service/guestbook-ui created
apps   Deployment  default    guestbook-ui  OutOfSync  Healthy        deployment.apps/guestbook-ui created

If you check your output, you will again see we are OutOfSync. In order to see what’s changed, you’ll use argocd app diff:

argocd app diff guestbook

===== apps/Deployment default/guestbook-ui ======
109c109
<   replicas: 1
--
>   replicas: 2

Run the sync command once again to deploy:

argocd app sync guestbook

Everything should be back healthy and in order!

Conclusion

As you can see, adopting GitOps is beneficial in many ways! Among other things, you gain faster and more frequent deployments, the ability to avoid configuration drift, and easier error handling. This is shown especially when coupled with Argo CD as it can help leverage your deployment process by performing commits automatically and can also execute rollbacks if there are any issues.

Argo CD is great by itself, but it is even better when connected to other Argo projects such as Argo Workflows, Rollouts, or Events. Combined with any of these, you can elevate your continuous deployment process.

Stop Using Branches for Deploying to Different GitOps Environments

Kostis Kapelonis — Fri, 17 Dec 2021 13:20:56 +0000

In our big guide for GitOps problems, we briefly explained (see points 3 and 4) how the current crop of GitOps tools don’t really cover the case of promotion between different environments or how even to model multi-cluster setups.

The question of “How do I promote a release to the next environment?” is becoming increasingly popular among organizations that want to adopt GitOps. And even though there are several possible answers, in this particular article I want to focus on what you should NOT do.

You should NOT use Git branches for modeling different environments. If the Git repository holding your configuration (manifests/templates in the case of Kubernetes) has branches named “staging”, “QA”, “Production” and so on, then you have fallen into a trap.

Let me repeat that. Using Git branches for modeling different environments is an anti-pattern. Don’t do it!

We will explore the following points on why this practice is an anti-pattern:

Using different Git branches for deployment environments is a relic of the past.
Pull requests and merges between different branches is problematic.
People are tempted to include environment specific code and create configuration drift.
As soon as you have a large number of environments, maintenance of all environments gets quickly out of hand.
The branch-per-environment model goes against the existing Kubernetes ecosystem.

Using branches for different environments should only be applied to legacy applications.

When I ask people why they chose to use Git branches for modelling different environments, almost always the answer is a variation of “we’ve always done it that way,” “it feels natural,” “this is what our developers know,” and so on.

And that is true. Most people are familiar with using branches for different environments. This practice was heavily popularized by the venerable Git-Flow model. But since the introduction of this model, things have changed a lot. Even the original author has placed a huge warning at the top advising people against adopting this model without understanding the repercussions.

The fact is that the Git-flow model:

Is focused on application source code and not environment configuration (let alone Kubernetes manifests).
Is best used when you need to support multiple versions of your application in production. This happens, but is not usually the case.

I am not going to talk too much about Git-flow here and its disadvantages because the present article is about GitOps environments and not application source code, but in summary, you should follow trunk-based development and use feature-flags if you need to support different features for different environments.

In the context of GitOps, the application source code and your configuration should also be in different Git repositories (one repository with just application code and one repository with Kubernetes manifests/templates). This means that your choice of branching for the application source code should not affect how branches are used in the environment repository that defines your environments.

When you adopt GitOps for your next project, you should start with a clean slate. Application developers can choose whatever branching strategy they want for the application source code (and even use Git-flow), but the configuration Git repository (that has all the Kubernetes manifests/templates) should NOT follow the branch-per-environment model.

Promotion is never a simple Git merge

Now that we know the history of using a branch-per-environment approach for deployments, we can talk about the actual disadvantages.

The main advantage of this approach is the argument that “Promotion is a simple git merge.” In theory, if you want to promote a release from QA to staging, you simply merge your QA branch into the staging branch. And when you are ready for production, you again merge the staging branch into the production branch, and you can be certain that all changes from staging have reached production.

Do you want to see what is different between production and staging? Just do a standard git diff between the two branches. Do you want to backport a configuration change from staging to QA? Again, a simple Git merge from the staging branch to qa will do the trick.

And if you want to place extra restrictions on promotions, you can use Pull Requests. So even though anybody could merge from qa to staging, if you want to merge something in the production branch, you can use a Pull Request and demand manual approval from all critical stakeholders.

This all sounds great in theory, and some trivial scenarios can actually work like this. But in practice, this is never the case. Promoting a release via a Git merge can suffer from merge conflicts, unwanted changes, and even the wrong order of changes.

As a simple example, let’s take this Kubernetes deployment that is currently sitting in the staging branch:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
spec:
  replicas: 15
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: backend
        image: my-app:2.2
        ports:
        - containerPort: 80

Your QA team has informed you that version 2.3 (which is in the QA branch) looks good, and it is ready to be moved to staging. You merge the QA branch to the staging branch, promoting the application and think that everything is good.

What you didn’t know is that somebody also changed the number of replicas in the QA branch to 2 because of some resource limitations. With your Git merge, you not only deployed 2.3 to staging, but you also scaled the replicas to 2 (instead of 15), and that is probably something that you don’t want.

You might argue that it would be easy to look at the replica count before merging, but remember that in a real scenario you have a large number of applications with a big number of manifests that are almost always templated (via Helm or Kustomize). So understanding what changes you want to bring and what to leave behind is not a trivial task.

And even if you do find changes that should not be promoted, you need to manually choose the “good” parts using git cherry-pick or other non-standard methods which are a far cry from the original “simple” Git merge.

But even if you are aware of all the changes that can be promoted, there are several cases where the order of promotion is not the same as the order of committing. As an example, the following 4 changes happen to the QA environment.

The ingress of the application is updated with an extra hostname.
Release 2.5 is promoted to the QA branch and all QA people start testing.
A problem is found with 2.5 and a Kubernetes configmap is fixed.
Resource limits are fine-tuned and committed to QA.

It is then decided that the ingress setting and the resource limits should move to the next environment (staging). But the QA team has not finished testing with the 2.5 release.

If you blindly merge the QA branch to the staging branch, you will get all 4 changes at once, including the promotion of 2.5.

To resolve this, again you need to use git cherry-pick or other manual methods.

There are even more complicated cases where the commits have dependencies between them, so even cherry-pick will not work.

In the example above, release 1.24 must be promoted to production. The problem is that one of the commits (the hotfix) contains a multitude of changes where some of them depend on another commit (the ingress config change) which itself cannot be moved to production (as it only applies only to staging). So even with cherry-picks, it is impossible to bring only the required changes from staging to production.

The end result is that promotion is never a simple Git merge. Most organizations will also have a large number of applications that go on a large number of clusters, composed by a large number of manifests. Manually choosing commits is a losing battle.

Configuration drift can be easily created by environment-specific changes

In theory, configuration drift should not be an issue with Git merges. If you make a change in staging and then merge that branch to production, then all your changes should transfer to the new environment.

In practice, however, things are different because most organizations only merge to one direction, and team members are easily tempted to change upstream environments and never back-port the changes to downstream environments.

In the classic example with 3 environments for QA, Staging, and Production, the direction of Git merges only goes to one direction. People merge the qa branch to staging and the staging branch to production. This means that changes only flow upwards.

QA -> Staging -> Production.

The classic scenario is that a quick configuration change is needed in production (a hotfix), and somebody applies the fix there. In the case of Kubernetes, this hotfix can be anything such as a change in an existing manifest or even a brand new manifest.

Now Production has a completely different configuration than staging. Next time a release is promoted from Staging to Production, Git will only notify you on what you will bring from Staging. The ad hoc change on production will never appear anywhere in the Pull Request.

This means that all subsequent deployments can fail, as production now has an undocumented change that will never be detected by any subsequent promotions.

In theory, you could backport such changes and merge periodically all commits from production to staging (and staging to QA). In practice, this never happens due to the reasons outlined in the previous point.

You can imagine that a large number of environments (and not just 3) further increases the problem.

In summary, promoting releases by Git merges does not solve configuration drift and in fact makes it even more problematic as teams are tempted to make ad hoc changes that are never promoted in sequence.

Managing different Git branches for a large number of environments is a losing battle

In all the previous examples, I only used 3 environments (qa-> staging-> production) to illustrate the disadvantages of branch-based environment promotion.

Depending on the size of your organization, you will have many more environments. If you factor in other dimensions such as geographical location, the number of environments can quickly skyrocket.

For example, let’s take a company that has 5 environments:

Load Testing
Integration testing
QA
Staging
Production

Then let’s assume that the last 3 environments are also deployed to EU, US, and Asia while the first 2 also have GPU and Non-GPU variations. This means that the company has a total of 13 environments. And this is for a single application.

If you follow a branch-based approach for your environments:

You need to have 13 long living Git branches at all times.
You need 19 pull requests for promoting a single change across all environments.
You have a two dimensional promotion matrix with 5 steps upwards and 2-3 steps outwards.
The possibilities for wrong merges, configuration drift and ad-hoc changes is now non-trivial across all environment combinations.

In the context of this example organization, all previous issues are now more prevalent.

The branch-per-environment model goes against Helm/Kustomize

Two of the most popular Kubernetes tools for describing applications are Helm and Kustomize. Let’s see how these two tools recommend modeling different environments.

For Helm, you need to create a generic chart that itself accepts parameters in the form of a values.yaml file. If you want to have different environments, you need multiple values files.

For Kustomize, you need to create a “base” configuration, and then each environment is modeled as an overlay that has its own folder:

In both cases, different environments are modeled with different folders/files. Helm and Kustomize know nothing about Git branches or Git merges or Pull Requests. They use just plain files.

Let me repeat that again: Both Helm and Kustomize use plain files for different environments and not Git branches. This should be a good hint on how to model different Kubernetes configurations using either of these tools.

If you introduce Git branches in the mix, you not only introduce extra complexity, but you also go against your own tooling.

The recommended way to promote releases in GitOps environments

Modeling different Kubernetes environments and promoting a release between them is a very common issue for all teams that adopt GitOps. Even though a very popular method is to use Git branches for each environment and assume that a promotion is a “simple” Git merge, we have seen in this article that this is an anti-pattern.

In the next article, we will see a better approach to model your different environments and promote releases between your Kubernetes cluster. The last point of the article (regarding Helm/Kustomize) should already give you a hint on how this approach works.

Stay tuned!

Using GitOps for Infrastructure and Applications With Crossplane and Argo CD

Kostis Kapelonis — Mon, 13 Dec 2021 13:43:05 +0000

If you have been following the Codefresh blog for a while, you might have noticed a common pattern in all the articles that talk about Kubernetes deployments. Almost all of them start with a Kubernetes cluster that is already there, and then the article explains how to deploy an application on top.

The reason for this simplification comes mainly from brevity and simplicity. We want to focus on the deployment part of the application and not its infrastructure just to make the article easier to follow. This is the obvious reason.

The hidden reason is that until recently infrastructure deployments were handled in a different manner than applications deployments. Especially in large enterprise companies, the skillset of people that deal with infrastructure and application can vary a lot as the tools of the trade are completely different.

For example, a lot of people that deal with infrastructure prefer to use Terraform templates, but employ Kustomize/Helm or other similar tools for application development. While this is a very valid solution, it doesn’t have to be this way.

Now with GitOps you can have a uniform way of dealing with infrastructure and applications.

GitOps and Terraform

If you are not familiar with GitOps, head over to https://opengitops.dev/ which is the official page of the GitOps working group. The principles of GitOps are the following:

The system is described in a declarative manner. (In practice, this means Kubernetes manifests.)
The definition of the system is versioned and audited. (In practice, it is stored in Git.)
A software agent automatically pulls the Git state and matches the platform state. (In practice, this means Flux/ArgoCD.)
The state is continuously reconciled. This means that any changes happening in Git should also be reflected in the system, as well as the opposite scenario.

If you have already worked with Terraform, you should already understand why it is difficult to apply GitOps to the Terraform CLI. While Terraform only has the first requirement (declarative format), it doesn’t satisfy the other three requirements around Git storage, automatic reconciliation, and two-way sync. As soon as Terraform finishes its job, it doesn’t interface with the system in any way. If you manually delete a Virtual Machine that was created by Terraform, vanilla Terraform doesn’t know about it. And regarding state, Terraform stores its own state which is completely different from the definition files in Git.

So at first glance, getting GitOps to work with infrastructure is a complicated task. You realize that you need to add something on top of vanilla Terraform in order to satisfy the GitOps requirements.

But there is a new kid on the block, and that is Crossplane!

GitOps and Crossplane

Crossplane has similar capabilities to Terraform (creating infrastructure) but with the following major differences:

Crossplane itself is a Kubernetes application. (The infrastructure it creates can be anything.)
Crossplane definitions are Kubernetes manifests.
You can either use manifests that describe resources in the common cloud providers or create your own resources.

As a simple example here is an EC2 instance describe in Crossplane:

apiVersion: ec2.aws.crossplane.io/v1alpha1
kind: Instance
metadata:
  name: sample-instance
spec:
  forProvider:
    region: us-east-1
    imageId: ami-0dc2d3e4c0f9ebd18
    securityGroupRefs:
      - name: sample-cluster-sg
    subnetIdRef:
      name: sample-subnet1  
  providerConfigRef:
    name: example

You can find more examples for Amazon, Google, and Azure. Crossplane supports several other providers, and of course you can add your own.

The important point here is that the file above is a standard Kubernetes manifest. You can:

Apply it with kubectl.
Verify it with manifest verification tools (kubeval or kube-linter).
Template it with Helm/Kustomize.
Use any of the tools in the Kubernetes ecosystem to read/manage/store it.

And since it is a standard manifest, you can of course store it in Git and manage it with ArgoCD for a Full GitOps workflow. The importance of this capability cannot be overstated.

If you combine ArgoCD and Crossplane, you have a full solution for following the GitOps principles with infrastructure and not just applications. Imagine if your ArgoCD dashboard contained this:

Isn’t that cool?

How ArgoCD and Crossplane work together

Crossplane offers an easy way to model your infrastructure as Kubernetes manifests. This is great on its own, but if you put ArgoCD in the mix, you essentially gain all the advantages of GitOps for your infrastructure.

You know exactly what infrastructure you have simply by looking at the Git repository.
You know exactly what was changed and when by looking at Git history.
The infrastructure state IS the same as the Git state. Terraform has its own state that is used a the single source of truth
You don’t need any external credentials any more.
You can easily rollback to a previous version of your infra with a git reset/git-revert. You completely avoid the dreaded configuration drift.

The last point is very critical. Terraform only knows what is in your infrastructure during deployment. If you make any manual changes afterwards (e.g. delete some infra), terraform knows absolutely nothing. You need to rerun terraform and pray that the correct action is taken. There are many terraform horror stories about incomplete/invalid states. Here is one of my favorite tfstate stories from Spotify.

ArgoCD will instantly detect any manual changes in your infrastructure, present you a diff, and even allow you to completely discard them if you want.

Essentially ArgoCD is agnostic as to what exactly is described by the Kubernetes manifests it manages. They can be plain Kubernetes applications, Virtual machines, Container registries, Load balancers, object storage, firewall rules, etc.

Create infrastructure and deploy to it using GitOps

Now that we saw that you can use ArgoCD and Crossplane together for managing infrastructure, we are now ready to treat applications and the platform they need in the same way.

This means that we can do the following:

Start from nothing.
Use Crossplane to create a Kubernetes cluster.
Commit the crossplane manifest and manage it with ArgoCD.
Use a standard manifest (e.g. deployment) to deploy an application to the cluster that was just created.
Commit that manifest in Git too. Like the application one, ArgoCD will manage it like any other Kubernetes manifest (even though it represents infrastructure).

Here is the whole workflow:

The Kubernetes cluster that Crossplane is running on is only used to bootstrap crossplane. It doesn’t get any production workloads on its own. For simple demos, this can be any cluster (even a local one running on your workstation).

The end result is that now you have a unified way to handle both infrastructure and applications right from Argo CD.

The process of changing either of them is exactly the same.

If you want to change infrastructure, you commit a change and ArgoCD takes care of it (with the help of Crossplane).
If you want to change your application, you commit a change and ArgoCD takes care of it.

They also both gain all the benefits of GitOps. If for example somebody changes the number of replicas in the deployment or tampers with the cluster nodes, ArgoCD will detect the manual changes automatically and give you the ability to discard them completely.

Why GitOps is the way forward

The whole point of DevOps is to make everything self-service and promote collaboration between developers and operators. Adopting Crossplane in a GitOps setting means that now both parties have a common language that they can communicate with.

There is no need anymore for separate workflows that are confusing to either of them. Adopting a common workflow for both infrastructure and applications is the embodiment of the DevOps spirit.

If you want to learn more about GitOps and how Codefresh has embraced it, check out the Codefresh DevOps platform for Argo. For more information about Crossplane, see the official site and the hosted solution by Upbound.

Using Codefresh with GKE Autopilot for native Kubernetes pipelines and GitOps deployment

Hannah — Mon, 06 Dec 2021 18:16:39 +0000

Several companies nowadays offer a cloud-native solution that manages Kubernetes applications and services. While these solutions seem easy at first glance, in reality, they still require manual maintenance.

As an example, an important decision for any Kubernetes cluster is the number of nodes and the autoscaling rules you define. You want to ensure that the size is just right and matches your workload, otherwise, this could become costly, unstable, and decrease the availability of your workload when you need it most!

GKE Autopilot Cluster

A solution to these problems is Google Kubernetes Engine (GKE) Autopilot from Google Cloud. GKE's cluster autoscaler adjusts the node pools based on whatever the demand is for your workload. For example, if the workload is high or low GKE Autopilot will scale depending on which and provide control on costs.

This allows you and your teams to no longer worry about maintaining your node pools or provisioning them - with Autopilot clusters these jobs are automatically done. This streamlined approach follows best practices for a cluster and workload setup, in addition to other best practices about security and observability. Allowing you to specify configurations needed for deployment to production and simplifying operations needed to manage an application’s infrastructure, nodes, and the control plane.

The Codefresh platform fully supports GKE Autopilot. By combining the two platforms you can easily:

Run CI/CD pipelines on your cluster to compile source code, run unit tests, perform security checks, etc.
Deploy your containers to your cluster by following the GitOps principles.

Deploying applications to your GKE Autopilot cluster following the GitOps principles

Codefresh GitOps Controller

Like any Kubernetes cluster integrations or the GKE Standard cluster, the GKE Autopilot Cluster integrates the same way with the Codefresh GitOps dashboard. The GitOps dashboard allows you to look at all your deployments in greater detail and instantly know the critical information such as:

What Git commit was deployed
Which cluster was affected
Who initiated the change
What features were shipped
Which pipelines took part in the deployment
What Kubernetes services were affected

To achieve this level of GitOps visibility, a GKE autopilot cluster can be the target of the Codefresh GitOps Controller, which is an agent installed in the cluster that monitors both the cluster and any defined Git repositories for changes. It installs an ArgoCD instance within the cluster and the associated GitOps Controller. This is done through the Codefresh CLI and can be authenticated with a Codefresh account, simply by creating an API token. This agent can also manage the runtime environment for a pipeline.

Once the GitOps controllers are installed, you can easily deploy any kind of application to your GKE Autopilot cluster using GitOps. This means that you can combine all the benefits of GitOps such as audibility, visibility, reduced downtime with the easy management of the GKE Autopilot capabilities.

Summary

GKE Autopilot cluster allows you to pursue a production-ready and scalable workflow for both your application and infrastructure systems. Its ability to configure cluster autoscaling and node auto-provisioning can improve resource utilization and reduce day-to-day challenges for organizations. These features reduce the operational cost of managing clusters and optimize them for production.

Google Cloud also provides a user-friendly experience, creating, configuring, and operating a Kubernetes cluster that you can integrate with the Codefresh DevOps Platform with a GitOps Controller, powered by ArgoCD.

Applied GitOps with Kustomize

Hannah — Wed, 01 Dec 2021 18:07:50 +0000

What is Kustomize?

Have you always wanted to have different settings between production and staging but never knew how? You can do this with Kustomize!

Kustomize is a CLI configuration manager for Kubernetes objects that leverage layering to preserve the base settings of the application. This is done by overlaying the declarative YAML artifacts to override default settings without actually making any changes to the original manifest.

Kustomize settings are defined in a kustomization.yaml file. Kustomize is also integrated with kubectl. With Kustomize, you can configure raw, template-free YAML files, which allows you to modify settings between deployment and production easily. This enables troubleshooting misconfigurations and keeps use-case-specific customization overrides intact.

Kustomize also allows you to scale easily by reusing a base file across all your environments (development, production, staging, etc.) and then overlay specifications for each.

Base Layer: This layer specifies the most common resources and original configuration.

Overlays Layer: This layer specifies use-case-specific resources by utilizing patches to override other kustomization files and Kubernetes manifests.

Overlays are what help us accomplish our goal by producing variants without templating.

Benefits of Kustomize

Kustomize offers some of the following benefits:

Reusability
With Kustomize you can reuse one of the base files across all environments (development, staging, production, etc.) and overlay specifications for each of those environments.
Quick Generation
Since Kustomize doesn't utilize templates, a standard YAML file can be used to declare configurations.
Debug Easily
Using a YAML file allows easy debugging, along with patches that isolate configurations, allowing you to pinpoint the root cause of performance issues quickly. You can also compare performance to the base configuration and other variations that are running.
Kubernetes Native Configuration
Kustomize understands Kubernetes resources and their fields and is not just a simple text templating solution like other tools.

See below an example of a Kustomize file structure including the base and overlays within an application. You can also reference a code-based demo project on GitHub.

The tree structure above is a simple example of how you can deploy a single application to 2 different environments (staging and production). Let's dig deeper into each directory and create an overlay.

base folder

The base folder holds common resources, such as the deployment.yaml, service.yaml, and configuration files. It contains the initial manifest and includes a namespace and label for the resources.

overlays folder

The overlays folder houses environment-specific overlays, which use patches to allow YAML files to be defined and overlaid on top of the base for any changes. Let's take a look at a couple of the environments within the overlays folder below that also includes the kustomization.yaml.

kustomization.yaml

Each directory contains a kustomization file, which is essentially a list of resources or manifests that describes how to generate or transform Kubernetes objects. There are multiple fields that can be added, and when this list is injected, the kustomization action can be referenced as an overlay that refers to the base.

Creating Overlays

Let's explore a simple example of how overlays work.
We'll make the following changes within our production and staging directories:

Within the staging overlay, we will enable a risky feature that is NOT enabled in production.

Within the production overlay, we'll assign a higher replica count.

We'll also ensure the web server from the cluster variants is different from one another.

overlays/staging/kustomization.yaml

In the staging directory, let's make a kustomization defining a new name prefix and different labels.

namePrefix: staging-
commonLabels:
 variant: staging
commonAnnotations:
  note: “Welcome to staging!”
bases:
- ../../base
patchesStrategicMerge:
- config-map.yaml

Staging patch

Add a configMap kustomization to change the server greeting from "Hello!" to "Kustomize rules!" We'll also enable the risky flag.

apiVersion: v1
kind: ConfigMap
metadata:
 name: the-map
data:
 altGreeting: “Kustomize rules!”
 enableRisky: “true”

overlays/production/kustomization.yaml

Within the production directory, we will make a kustomization with a different name prefix and label.

namePrefix: production-
commonLabels:
 variant: production
commonAnnotations:
  note: “Welcome to production!”
bases:
- ../../base
patchesStrategicMerge:
- deployment.yaml

Production patch

We'll make a production patch that will increase the replica count.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: the-deployment
spec:
 replicas: 10

Now, we can compare these overlays - the kustomizations and patches are required to create noticeable differences between staging and production variants within the Kubernetes cluster.

Based on the changes above, the output would look something like this:

<   altGreeting: Kustomize rules!
<   enableRisky: "true"
---
>   altGreeting: Hello!
>   enableRisky: "false"

<     note: Welcome, I am staging!
---
>     note: Welcome, I am production!

<     variant: staging
---
>     variant: production

You can now see the difference between the staging and production overlays.
Overlays contain a kustomization.yaml, and can also include manifests as new or additional resources, or to patch resources. The kustomization file is what defines how the overlays should be applied to the base and this is what we refer to as a variant.

Each time a change is made to an application, like the example above - it is the overlays that are doing the heavy lifting.

So, now that you have learned a bit about Kustomize, you might be wondering how you can apply the GitOps workflow to it. Below we'll explain more about GitOps and how you can apply it to your Kustomize project and deploy it.

GitOps works with all your existing tools

Now, let's learn more about how to apply GitOps to your Kustomize application deployment!

GitOps is a paradigm that incorporates best practices applied to both an application development workflow and the infrastructure of a system. This is done to empower organizations and developers to operate their systems from a single source of truth enabled by Git.
A couple of key aspects of GitOps that most aren't as familiar with and are crucial are the:

GitOps controller

The controller reads the declarative configuration and uses the reconciliation loops to converge the desired end state. This controller detects the differences between states and makes the necessary changes to maintain the desired state.

Automation reconciliation loop

The concept of the reconciliation loop is used to keep the state as defined within a manifest. This is such an important principle for the GitOps paradigm because these loops are what drive the entire cluster to the desired state in Git.

Now that you have a basic understanding of what both Kustomize and GitOps are and what some of their capabilities are, perhaps you've considered applying GitOps to your existing or future application and infrastructure workflow.

We'll explore 2 approaches on how to use Kustomize with or without GitOps:

Using only Kustomize

Using Kustomize with ArgoCD

What is ArgoCD?

ArgoCD is a GitOps controller specifically created for Kubernetes deployments. It supports a variety of configuration management tools like Helm (take a look at our documentation for more information), Ksonnet, Kustomize, etc. The core component is the Application Controller, which continuously monitors any running applications and compares the live state against the desired state defined in a Git repository.

If a deployed application whose live state drifts from the target state, ArgoCD is then considered OutOfSync. It then provides reports and visualizations to identify these changes and can provide automation when any modifications are made so that the target environments reflect the desired state of a system in Git.

If you'd like to explore these different deployment approaches with Kustomize from a code-based perspective, here is an example project that you can follow along with on GitHub. However, we will explore both of these approaches below with some more details.

Approach #1: Deploying only with Kustomize

If you want to use Kustomize, but your organization isn't ready to implement GitOps with your workflow, you can still use Kustomize on its own.

Install Kustomize

First, install Kustomize and this can be done either by utilizing kubectl version 1.14 or later, otherwise you can install based on your operating system and reference the Kustomize documentation.

Create a base directory

Next, in order to deploy an application with Kustomize, you need a kustomization.yaml file, and this is added to the base directory.

The kustomization.yaml file specifies what resources to manage due to the complexity of multiple resource types and different environments when handling configuration files for Kubernetes.

Within the base directory, there is also a service and deployment resource.

Create an Overlays directory

Next, you'll want to create an overlays directory, and this directory is what allows you to kustomize the base and apply any changes with a patch to modify a resource.

The overlays still use the same resources as the base but may vary in the number of replicas in a deployment, the CPU for a specific pod, or the data source used in the ConfigMap, etc.

Within our example, we have a staging and production overlay. These overlay directories contain the kustomizations and patches previously mentioned that are required to create distinct staging and production variants in a cluster. However, with Kustomize you can use the overlays for anything needed to organize environments, whether it's based on location: USA, Asia, Europe or internal/external, team A/team B, etc.

Kustomize is essentially an overlay-based engine that functions by finding and replacing specific sections in the manifest and replacing it with required fields and values. These values are then merged and deployed!

Create a namespace for specific environments

kubectl create ns staging

kubectl create ns production

Build and Deploy to environments

You can then apply the overlays to your cluster and deploy with the command:

kubectl apply -k overlays/staging

kubectl apply -k overlays/production

You are now successfully using Kustomize to manage your Kubernetes configurations and deploy your application to production and staging environments!

Approach #2: Deploying using Kustomize with GitOps

In the previous approach, you learned how to deploy an application using only Kustomize, so let's see how it fits into ArgoCD and how it can be used in a GitOps workflow.

ArgoCD supports Kustomize and has the ability to read a kustomization.yaml file to enable deployment with Kustomize and allow ArgoCD to manage the state of the YAML files.

ArgoCD monitors the resources within the git repository for any changes, ensuring that the live state of your system matches the desired state. Each time customization is made, ArgoCD detects those modifications and updates the deployment.

So, let's begin walking through the process to deploy a Kustomize project using ArgoCD!

Install ArgoCD and access a Kubernetes Cluster

First, we need to ensure the Kubernetes cluster is set up and you are logged into ArgoCD so that these resources are provided and can be deployed. You can use any Kubernetes cluster and install the argocd CLI.

Once you've accessed the argocd CLI you can access the ArgoCD server and log in to the ArgoCD UI. However, if you're more of a terminal fan, you can also deploy the application through the argocd CLI. Feel free to reference the demo application that will walk you through the deployment process through your terminal.

Create an ArgoCD application

Now, we can set up the Kustomize Project. Similar to using Helm with GitOps, we will approach this deployment the same way within the UI by creating an ArgoCD application.

Let's begin! First, click on the +NEW APP and include the name of the ArgoCD application, select the default project, and enable the Automatic SYNC POLICY.

Next, when adding your Kustomize project it helps to include a specific Path to segregate the application manifests inside the Git repository. We can then ask ArgoCD to only read the specific directories in the repository and read manifests within that path.

Then, within the Destination section, you need to provide the destination of the Kubernetes cluster details. Also, make sure to click on the checkbox for auto-create namespace when adding the input field value, or you can add a custom namespace you've created prior.

ArgoCD will read the kustomization.yaml file in the path you provided and prompt you to allow override with different values.

Synchronize Application and Deploy

Assuming you enabled auto-synchronization when creating your ArgoCD application, it will read the parameters and the Kubernetes manifests. Then, once the manifests are applied, you can review the application health and resources you deployed.

If your application has an error when trying to synchronize, you can execute the argocd history command, allowing you to view the application deployment history to identify a possible error:

argocd app history argocd_app_name

If you need to rollback you can do so by executing this command:

argocd app rollback argocd_app_name history_id

These commands leverage a faster and more secure deployment by enabling the tracking from the Git repository. This allows you to track the active Kubernetes resources and events. These actions can also be done within the UI.

Once the application is healthy and synchronized, each time you create a new kustomization.yaml file and the file changes, ArgoCD will be able to detect those changes and make updates to your deployment. You can even mention specific Kustomize tags and set up custom build options for your Kustomize build.

Summary

Kustomize allows you to use different configurations of a base Kubernetes manifest. Within this post, we've covered the Kustomize basics and how to deploy using just Kustomize and deploying with GitOps. This allows you to leverage the power of Kustomize to define the Kubernetes files without using a templating system.

To start deploying your Kustomize application, sign up for a Codefresh account today to apply GitOps to your deployment process!

Using Helm with GitOps

Hannah — Thu, 30 Sep 2021 15:15:45 +0000

This is the first of many posts highlighting GitOps topics that we’ll be exploring. Within this post, we will explore Helm, a tool used for Kubernetes package management, that also provides templating. Helm provides utilities that assist Kubernetes application deployment.

In order to better understand how Helm charts are mapped to Kubernetes manifests, we’ll explain more details below and how to use Helm with and without GitOps.

What is Kubernetes?

Kubernetes is an orchestrator for containers that allows you to automate scheduling, deployments, networking, scaling, and health monitoring for the containers.

Kubernetes was needed because we increased the usage of the following:

Microservices
Containers

This was difficult to manage with scripts and self-made tools and caused the need for orchestration technology like Kubernetes. It provides features like scalability, disaster recovery, and less downtime.

What is Helm?

Helm is a package manager for Kubernetes. It’s a convenient way for packaging collections of YAML files with a Helm chart for the Kubernetes application and allowing distribution with a Helm repository.

Maintaining Kubernetes manifests is time-consuming and tedious and this is why Helm is helpful!

Helm Charts

Helm Charts are deployable units for Kubernetes applications. These charts are a collection of files inside a directory. This directory is the name of the Helm chart and consists of a self-descriptor file, YAML file, and one or more Kubernetes manifests. Helm charts are typically written in the Go template language. Charts are created as such files, describing a related set of Kubernetes resources. Here’s an example of a Helm chart directory and it’s layout:

This particular directory contains a Chart.yaml file and this is where the global variables, versions, and descriptions are stored. Then, the templates directory is what contains the YAML files for Kubernetes, otherwise known as the Kubernetes manifests.

Files such as the deployment, service, and ingress files contain variables from the values.yaml file when the chart is deployed. The _helpers.tpl incorporates helpful functions for variable calculations.

You can then share the Helm chart to increase easy reusability for others to use. Sharing is done by storing the chart to a Helm repository. This repository can then be shared with others to deploy the application with the chart.

Helm Repositories

Helm supports a chart repository service that can be used to store the Helm charts. You can use any web server host or source code host for the Helm repository.

The repository has an index.yaml file that contains metadata about the package, including the Chart.yaml file. The index will contain information about each Helm chart in the chart repository. Then the server can serve your index and charts or the packages can be stored in the repository for shareable access.

Helm Release

Each install or upgrade will create a Helm release. A Helm release is a running instance of your Helm chart running within a Kubernetes cluster or a namespace. It’s essentially an instance of a versioned, templated chart. It’s also possible to have multiple releases of the same chart in a single cluster or namespace since the chart is self-contained. You can also roll back a Helm release to a previous version in case there are any failures.

Assuming you’re a developer with an existing cluster, perhaps you’d like to share your Helm application with an external vendor. Let's recap the Helm workflow and release process:

To share an application with others, you need to create a Helm chart. The chart is a package that contains templates for a set of resources necessary for the application. The template uses variables applied to the Values.yaml file when the manifest is created and describes how to configure the resources.
The Helm charts are then hosted within a repository that can then be downloaded or accessed from a server. This chart will contain all the necessary resource definitions needed for the developer to run an application.
Now that you have access to the cluster, you can add new features or bug fixes to the application and update the Helm chart. Helm offers useful tools to manage your releases. You can upgrade the chart and create a new deployment. In Helm a deployed instance of your application is referred to as the release.
You can now deploy your packaged application to the cluster.

Within this workflow we identified how Helm packages files and deploys them to a cluster. So, how do you apply GitOps to the workflow above? Let’s explore this below...

What is GitOps?

GitOps is a paradigm that incorporates best practices applied to the application development workflow all the way to the operating infrastructure of a system.

Some benefits of GitOps are:

Deploying faster and more often
Easier and quicker error handling and recovery
Self documenting deployments
Increased developer productivity and an enhanced experience for teams
Greater visibility on the lifecycle of developed features

These benefits make it easier to handle the applications and allow teams to deliver quality software faster.

GitOps relies on Git as the single source of truth for declarative configuration and active reconciliation. By adopting the GitOps methodology it provides transparency between the application configuration deployed in a cluster and the one residing in Git.

A GitOps tool that follows this approach of a Git-based workflow is Argo CD. It’s a continuous delivery tool for Kubernetes that is essentially a GitOps controller that does two-way synchronization. Argo continuously monitors running applications and compares the live state against the desired state in Git and applies it to the cluster. In addition, it also monitors the container registry for new images and updates the workload definitions based on the deployment policies.

We will mention Argo CD again in this post and can assume that it’s already been installed and implemented within our workflow!

GitOps works with all of your existing tools

Now that you have a good understanding of what Helm is and it’s capabilities, perhaps you’ve considered applying GitOps to your existing or future applications.

Let’s explore the various approaches of how to use GitOps with or without Helm:

Using only Helm (without GitOps)
Using GitOps (without Helm)
Using Helm with GitOps

If you’d like to explore these different approaches from a code-base perspective, here is an example project that you should check out on GitHub that digs deeper into how you can:

Install an application with Helm and deploy locally
Install an application with Argo CD and deploy locally both within the UI and command line

If not, then we’ll be sure to highlight these approaches below!

Approach #1: Using only Helm without GitOps

If you want to use Helm but your organization isn’t quite ready to implement the GitOps workflow, that’s perfectly fine. Helm was created before the GitOps methodology came to fruition, but can still work with GitOps (as we will see in the latter point).

The workflow allows you to search through Helm repositories for charts and install them to clusters, creating releases. This process enables execution of application deployments on a cluster. The maintenance of YAML manifests for Kubernetes objects is still the same as before. Helm releases are what keeps track of the version history of each chart installation and change and still allows rolling back to the previous cluster version. When installing a chart it creates a release of the new package and it’s the Helm release that deploys the Helm application.

When using Helm there are some powerful commands you will find useful, please see below.

To install a new package:

helm install <release_name> <name_of_chart_you_want_to_install>

To view currently deployed releases:

helm list

helm ls

To view revision numbers for a particular release:

helm history <release_name>

To learn more details about these commands and how to use them, reference the example project in GitHub.

Approach #2: Using GitOps without Helm

If you’re sold on the idea of GitOps, there are alternative tools that can be used if you’re unable to use Helm or choose not to.

You can use ArgoCD on its own. It is pretty flexible and can work with other templating solutions or even plain manifests. Another templating tool that can be used instead of Helm is Kustomize. This tool is built into kubectl and is native to Kubernetes. It allows you to customize Kubernetes configurations using only the Kubernetes API resource files.

Now, let’s explore our third approach about how to implement GitOps using Helm and Argo CD...

Approach #3: Using Helm with GitOps

Now that you’ve seen how GitOps can be applied without Helm, let’s explore how you can use it with Helm.

First, the Helm chart and any application changes must be committed in Git prior to being applied to the cluster. Meaning all of your workload definitions in a YAML format, the Helm charts, and any other Kubernetes custom resources that define the cluster’s desired state must be committed to the repo. This way rollbacks and logs are accessible and the state of production can be restored easily.

Within Argo CD you can connect the Git repository using HTTPS and add the URL directly. This then allows you to enable auto-synchronization to automatically synchronize the cluster to the desired state in the Git repository. Once synced successfully, the application status is recognized as Healthy. Otherwise, if the application is OutOfSync, you can rollback or view the release history to resolve the issue.

Here’s an example of a Healthy application in the Argo CD UI.

When the application is running you can view its resource components, logs, events, and the health status.

For good measure, here’s a visual of the Argo CD UI exemplifying an OutOfSync application when the live state deviates from the target state.

If your application had an error or was OutOfSync like the example above, you could execute the argo history command that allows you to view the application deployment history:

argocd app history <argocd_app_name>

Then, if you need to rollback the deployment you can do so by executing this command:

argocd app rollback <argocd_app_name> <history_id>

Essentially, these commands leverage a faster and more secure deployment, by enabling the tracking from the Git repository. It also allows you to track the active Kubernetes resources and events.

Therefore, using source control like this is what classifies this application deployment as GitOps!

Something important to note is that Argo CD provides native support for Helm, meaning you can directly connect a packaged Helm chart and Argo CD will monitor it for new versions. When this takes place the Helm chart will no longer function as a Helm chart and instead, is rendered with the Helm template when Argo is installed, using the Argo CD application manifest.

Argo CD then deploys and monitors the application components until both states are identical. The application is no longer a Helm application and is now recognized as an Argo app and can only operate by Argo CD. Hence if you execute the helm list command, you should no longer be able to view your helm release because the Helm metadata no longer exists.

Here’s an example of the command output. As you can see, the Argo CD application is NOT detected as a Helm application.

Summary

Helm is a powerful tool that is used often with Kubernetes deployments and provides tracking for each release. This helps ensure reliable and quick deployments for development teams.

So, if you’re completely new to Helm, I suggest referring to our documentation to learn more about it, here:

We'll discuss additional tools in our next post that can help you and your team continue implementing a GitOps process to your development and infrastructure systems.