Forem: Raul Castellanos

Managing database migrations safely in high replicated k8s deployment.

Raul Castellanos — Sun, 13 Nov 2022 12:49:04 +0000

So, you want to run migrations in a cloud native application running on a Kubernetes cluster, and don't die trying huh!

Well you're in the right place!!

After I break some applications in terms of database migrations in a multi replica and concurrent deployment process, I want to give you some advices based on my faults, on how you can run you migrations in a safely way, with native k8s specs and without hacks of any kind. (no helm, no external deployers, pure and plaink8s process well orchestrated)

The Problem

Is very common, modern application evolves faster, new features arise from product to satisfy the final user, and with every new deploy is too common the need to alter your database in some form and you have, many tools to allow you to manage the execution of the migrations against your database, BUT, not when they occur.

If you have an application pod, let say with 4 replicas, and you deploy it, the all 4 will try to run the migrations at the same time potentially causing data corruption and data loss, and nobody wants that.

When to run migrations `(the workflow)`

In the old-way of run migrations we used to have something like this:

Put the application in maintenance mode (divert traffic to a special page)
Run database migrations
Deploy new base code
Disable maintenance mode on application

Obviously, this isn't acceptable approach if you want to achieve zero-downtime deployments in the actual always-on world, we need to achieve (at leats) the following steps to assure that migrations and application run in a safety way.

Run migrations while the old version of the application is still running, and do the rolling update "only" when migrations are successfully run.

Something like this:

`Job`, `InitContainer` and `RollingUpdates`

So after choose our strategy to run migrations on k8s, we need to write our manifests in order to accomplish the defined workflow.

First, the migrations job itself:

apiVersion: batch/v1
kind: Job
metadata:
  name: availability-api-migrations
spec:
  ttlSecondsAfterFinished: 60
  backoffLimit: 1
  template:
    spec:
      containers:
        - name: migrations
          image: availability-api-migrations
          command:
            - '/bin/sh'
            - '-c'
            - ' bin/console doctrine:migrations:migrate --no-interaction -v'
          envFrom:
            - secretRef:
                name: protected-credentials-from-vault
      restartPolicy: Never

Now in the deployment manifest of the application we need to defined 2 things very important to allow our workflow work as expected.

The rolling update strategy
the init container and command to forbid deployment init until migrations are done.

apiVersion: apps/v1
kind: Deployment
...
spec:
  ...
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
   template:
      spec:
        initContainers:
          - name: wait-for-migrations-job
            image: bitnami/kubectl:1.25
            command:
              - 'kubectl'
            args:
              - 'wait'
              - '--for=condition=complete'
              - '--timeout=600s'
              - 'job/availability-api-migrations'

What is the meaning of that snippet above:

RollingUpdate strategy:
- The rolling update allow us, to define the strategy on how many pod replicas we want to update with the new code at a time (you can also choose between other k8s strategies like recreate, blue/green or canary)
InitContainer migration job watcher
- Now we want, to forbid in some way that the rollout deployment begins until the migrations job was finished in completed status. Fortunately the kubectl cli tools allow us to ask and wait the status of k8s component, we use that advantage of the kubelete api to use here, and "block" in some way the deployment

$kubectl wait --for-condition=complete --timeout=600 job/availability-api-migrations

We wait for the job availability-api-migrations status complete with a timeout o five minutes, the kubectl will ask permanently until timeout was reach. Obviously if the migration job finish in there milliseconds, automatically the wait loop ends and allow the deploy to begin otherwise, the deploy will fail. The timeout is the MAX allowed time to wait for ask for complete status on a job.

At least with this we can assure that data consistency is preserve, but it's importan follow some guidelines in terms on how to write and deploy migrations in you applications, below we'll cover that.

Safety recommendations

Write your migrations always thinking in a fast execution, if you expect to run migrations that took more than 5 minutes, considering ask for a maintenance window to restrict traffic access to the application.
Do your migrations always thinking in retro compatibility of your current code running on production.
- For example, do not alter a table adding a column NOT NULLABLE or without DEFAULT value.
If you need to add a column and delete other, the best strategy to follow is to do 2 separates deployments.
- Run the first deploy adding the column, and validate that all is running smoothly in production.
- Run a new deployment only deleting the old column from the database.

Support me

If you like what you just read and you find it valuable, then you can buy me a coffee by clicking the link in the image below, it would be appreciated.

How to build a CI/CD workflow with Skaffold for your application (Part III)

Raul Castellanos — Wed, 09 Nov 2022 09:06:04 +0000

Let's recap: `The Workflow`

This is the workflow so far:

📣 You can check how to get to this point in the firsts two delivery of the series.

Gitlab `K8s agent` and Security

This main part in the integration of k8s and Gitlab with the Gitlab K8s Agent, is, in my experience, the best and easy way I find to integrate K8s with a DevOps platform like Gitlab.

Let's recap some steps

You need to add an Agent and them run a helm chart into your cluster to allow the secure communication between both.
the agent can be configured in 2 ways:
- CI_ACCESS: Allow access from the project repository pipeline to the cluster and then you are in charge to manage how to deploy in the cluster.
- GITOPS_ACCESS: This allow a full gitops flow like ArgoCD for example, updating your cluster in a pull based way in sync with the main branch of the repository.

In my case a use the first one CI_ACCESS since I want to manage in a more granular way, the whole process with skaffold, so mi configuration is way simpler

I have 2 repositories in a application group, one for the micro service itself and one for the agent (the agent also could be put in the micro service repository, but if you want more granular access or share the agent/cluster between applications of the same stack, this is the best way)

So in the K8s-agents repository, we only have the declarative config.yaml file for every agent that we want to create (for this example I have 2, one for lower-envs/runner and one for production since they are 2 different clusters)

And in the config itself, I give access to all the projects that I want to use in the cluster.

And last, bit no least, the need to link the agent with the cluster, for that, you should go to the k8s project, Kubernetes Cluster menu, and there you will see an interface where you will receive the instructions on how to link agent/cluster via a helm chart to be installed in the cluster.

After that, you cluster and gitlab instance are now linked, and you applications can use the cluster as Kubernetes-Executor runners and also for dynamic review environments (aka dynamic QA instances to say so)

Deployment and Safety Recommendations for `K8s Agents`

To restrict access to your cluster, you can use impersonation. To specify impersonations, use the access_as attribute in your Agent's configuration file and use K8s RBAC rules to manage impersonated account permissions.

You can impersonate:

The Agent itself (default) = The CI job that accesses the cluster
A specific user or system account defined within the cluster

Impersonation give some benefits in terms of security:

Allows you to leverage your K8s authorisation capabilities to limit the permissions of what can be done with the CI/CD tunnel on your running cluster
Lowers the risk of providing unlimited access to your K8s cluster with the CI/CD tunnel
Segments fine-grained permissions with the CI/CD tunnel at the project or group level
Controls permissions with the CI/CD tunnel at the username or service account

Provisioning cluster with `terraform`

As I said, the main goal of this tutorial is try to get the same tooling for local development, pipeline and deployment, but (always got a but), we have 2 sets of the terraform configuration instructions, of example for local development I want, as developer, to get as much of the observability tools that I have on production, in case I need to test metrics, build dashboards on grafana, etc, but without the complexity of production infrastructure architecture.

So in this case, we can give the application diagram for the first part here, as recap of 'how our local development stacklooks like, and how to achieved withterraform `.

So, for this application structure, i want to get in my local environment the cluster and it's 'pre-requisites' for my architecture, understood as pre-requisites all the others components inside the cluster that no belongs to the application itself (monitoring stack, traefik, cert-manager, etc).

So for that, I write simple modules to install that dependencies inside the local cluster, and get them available to use when a run my application locally with skaffold.

My structure for infrastructure folder looks like this:

Every module has inside their main.tf configuration file setting the desired state for my cluster after it's applied.

Lets take a look for one of this module (prometheus) main file:

`
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.13.1"
}
helm = {
source = "hashicorp/helm"
version = ">= 2.7.0"
}
kubectl = {
source = "gavinbunney/kubectl"
version = ">= 1.14.0"
}
}
}

resource "kubernetes_namespace_v1" "monitoring_namespace" {
metadata {
name = var.monitoring_stack_namespace
}
}

resource "helm_release" "prometheus_stack" {
name = var.monitoring_stack_prometheus_name
repository = "https://prometheus-community.github.io/helm-charts"
chart = "prometheus"
version = var.monitoring_stack_prometheus_version_number
namespace = var.monitoring_stack_namespace
create_namespace = false

values = [
file("${path.module}/manifests/prometheus-override-values.yaml")
]

depends_on = [
kubernetes_namespace_v1.monitoring_namespace
]
}

resource "kubectl_manifest" "prometheus_stack_ingress" {
yaml_body = file("${path.module}/manifests/prometheus-ingress.yaml")

depends_on = [
helm_release.prometheus_stack
]
}

And then, in the root main.tf configuration file, you can wrap as many modules as you want, for my case, with my 4 modules was enough (prometheus, traefik, cert-manager, grafana)

`
module "cert_manager_stack" {
source = "./module/cert-manager"
}

module "traefik_stack" {
source = "./module/traefik"

depends_on = [
module.cert_manager_stack
]
}

module "prometheus_stack" {
source = "./module/prometheus"

depends_on = [
module.traefik_stack
]
}

module "grafana_stack" {
source = "./module/grafana"

depends_on = [
module.prometheus_stack
]
}

This also, has a handy target in our Makefile, allowing developers and operators, easily setup and remove cluster pre-requisites

Setup the pre-requisites took about 3 minutes, but if something that you need to do, time to time, you can setup your cluster today, work on your feature for some days, and then shutdown.

After all this, you will have a fully functional Local-To-Prod pipeline. (if you need to see how the Gitlab CI file looks like, is in the second part of this series)

This is the las delivery of the series, but now I'll write about the others tools that i use to address different challenges in my day to day work.

If you are interested the next topics I'll write on are:

Managing database migrations at scale in Kubernetes for PHP application with symfony/migrations component
Istio, Cert-Manager and Let's Encrypt: Secured your k8 clusters communication with automated generation and provisioning of SSL Certificates
Internal Developer Platform: A modern way to run engineering teams.
The Digital War Room or how to get observability for Engineering Managers across applications and teams.

Support me

If you like what you just read and you find it valuable, then you can buy me a coffee by clicking the link in the image below, it would be appreciated.

How to build a CI/CD workflow with Skaffold for your application (Part II)

Raul Castellanos — Mon, 07 Nov 2022 08:45:42 +0000

Lets recap the `Workflow`

As you remind - and if not, you can read the first release of this tutorial 😅 - my main idea is to implement one tool - skaffold - as building block for my CI/CD workflow, who should be managed from a single makefile as entrypoint for local development and pipelines - on gitlab - and all this should be deployed in a K8s cluster in GCP

And also, for simplicity of this tutorial, we use the image and artefact repository in the same gitlab SAAS, but you can use whatever you want for this task (Amazon S3, Docker Registry, Private Registries, Azure Object Storage, etc)

This is the workflow so far:

📣 The tool setup and local workflow was covered in the first delivery of this tutorial.

The Makefile

As said, the makefile, is, the main entrypoint for commands to be executed by developers when they do development work locally, and for gitlab pipelines in their different stages (must vary on your implementation), and that makefile commands mostly are wraps for skaffold ones, with the difference, that i need to pass dynamic values to those stages to work I expect to, so it's better to wrap then in a makefile target that received that dynamic params and then run the 'skaffold' command itself. (later you will now why)

As for the time I was writing this article, my targets are:

Hopefully for local development we have an unique command since skaffold run do the full pipeline cycle (build, test, deploy, hot reload) for development , so, why then i need to wrap this single command in a makefile target?, mostly because I need that this work no matter what type/kind of local cluster technology the developer use (docker-desktop, Minikube, etc) and no matter what OS developer machine was (MacOS, *Nix) and for that, I need to pass the kube-context parameter to skaffold that in my case is docker-desktop (since docker desktop already bring me a pre installed k8s cluster for local development and that free me to the need to install manually a cluster in my machine - a win for docker-desktop here -.

So you will encounter that the majority of the makefile wraps, come in the form of rehuse the same command in multiple stages (pipelines) based on the received parameter in the make execution, and also, generating random seeds to prefix namespaces (because you will have more than one developer working in the same code base at once) and I want to avoid collision between gitlab pipelines and deploys in lower environments when N developers are working in the same code base.

We must focused in pipeline target ones (all of then with skaffold tool), since the infrastructure ones, related to install prerequisites in local cluster to comply with the architecture design, isn't cover in this series, but in the another series that 'll wrote in next weeks)

Here is how my makefile looks at the time of writing this article, and for those stages (because in technology everything evolves quickly I always remind this 😅)

#------ Development and Pipeline targets ----------#
run: ## DEVELOPMENT[skaffold]: Up and running stack in development mode with hot reloading in the local machine
    @$(MAKE) _requirements
    @skaffold dev -f $(DEPLOY_DIR)/skaffold.yaml -p development -n $(PROJECT_NAME) --no-prune=false --cache-artifacts=false
unit: ## DEVELOPMENT[skaffold]: build, deploy and run unit tests => FOR PIPELINE: `make unit profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=unit PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
integration: ## DEVELOPMENT[skaffold]: build, deploy and run integration tests => FOR PIPELINE: `make integration profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=integration PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
functional: ## DEVELOPMENT[skaffold]: build, deploy and run functional tests => FOR PIPELINE: `make functional profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=functional PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
build: ## PIPELINE[skaffold]: build and push images to registry => `make build tag=1.0.0|71dcab00 kube_context=docker-desktop`
    @skaffold build -f $(DEPLOY_DIR)/skaffold.yaml -p production -t $(tag) --kube-context=$(kube_context) --file-output=pipeline-artifacts.json
render: ## PIPELINE[skaffold]: render manifests and push to artifact registry => `make render namespace=availability tag=1.0.0|71dcab00`
    @skaffold render -f $(DEPLOY_DIR)/skaffold.yaml -p production -n $(PROJECT_NAME) -a pipeline-artifacts.json -o $(PROJECT_NAME)-api-$(tag)-production.yaml
deploy: ## PIPELINE[skaffold]: apply hydrated manifests to desired namespace on cluster `make deploy tag=1.0.0|71dcab00 profile=production namespace=availability kube_context=docker-desktop`
    @kubectl create namespace $(namespace) --context=$(kube_context)
    @skaffold apply -f $(DEPLOY_DIR)/skaffold.yaml -p $(profile) -n $(namespace) --kube-context=$(kube_context) --status-check=true $(PROJECT_NAME)-api-$(tag)-production.yaml || $(MAKE) _cleanup KUBE_CONTEXT=$(kube_context) PROFILE=$(profile) NAMESPACE=$(namespace)

As you can see, all targets are warps for skaffold, but, allowing me to pass dynamic data, context and profiles (we'll look this in the skaffold file explanation below), the complete file is more larger, but with this snippet you can get an idea on how you can build something like that.

The `skaffold` file

The main workflow orchestrator has a main config file, when we can define, how the application should be builded, tested, render their manifest and deployed, so it's a vertebral part of this strategy, and now I'll explain you mine.

I have two skaffold profiles: one called development and the other production, and a common part share between them like tag strategy, deploy strategy, and both of them use the sameDockerfileto build their images pointing to the correcttarget`. (You can use separate Dockerfiles, but in my case I want to maintain as simple as possible because the difference between 2 images dependencies are minimum)

the skaffold fiel lives inside my deploy folder (do you remember my file organisation? You can re-checked in the first delivery of this series), and all the deployment related files are store there.

As you can see the skaffold.yaml file is in the root of the deploy folder since is my main workflow orchestrator, in the other folder we have the main k8s manifests and inside overlays we have the yaml patch's for every profile (most of the cases the same as an environment) that we want to declare.

The `.gitlab-ci.yml` Pipeline

Since I use trunk base development strategy in micro services, we need to design 2 flows of pipelines, one for features branches and one for main branch, additionally to that, to be able to reach production we first need to tag a commit, so that's the real trigger for Go-To-Prod Pipeline.

Feature Branches Workflow (trigger by merge_request commit):

In this stage, the developer needs to be able to:

Run all tests suites
Build and Image with production dependencies
Deploy to a dynamic namespace in the same cluster where pipeline runs (a Dynamic QA to say so, allowing to every developer to have "their own" QA server while their are working on a feature)
Destroy the dynamic deployment (remove review app and namespace)

Looks like this:

Also we use gitlab environments to see in the Gitlab UI "what's is deployed in what environment", and also to visualise production and stages latest deployment status and artefacts.

So, when a new "dynamic QA" environment is deployed, you'll see this in environment page of your project:

And also on the MR View on Gitlab you'll see in what "review" environment is deployed that MR.

Main Branch Workflow (trigger by a TAG):

In this stage, the developer needs to be able to:

Run all tests suites
Build and Image with production dependencies
Create a Release Package (this a gitlab feature like Github to display release contents in a special page on gitlab)
Deploy to Production

Looks like this:

The main workflow has 3 stages more, including deploying to staging cluster/namespace, but more important has the creation of the release package and the deployment to production.

The only manual job is deploy to production.

Gitlab Release

Production Environment after Deployment

All the best of all, all this is DONE automatically via the automated Pipeline on Gitlab. You can view the skeleton of the pipeline in this Snippet:

https://gitlab.com/playground-arena/api-symfony-roadrunner-cqrs/-/snippets/2448780

BONUS: `Kaniko` image Builder

Since version 1.24, Kubernetes was moving away from dockershim as container runtime, so I want a solution that:

Allow me to build container inside a K8s cluster
Continue to writing Dockerfiles as today
Not need to share or mount a socket into a pod (that's the main security reason to not use docker)

So I discover the last year Kaniko, this is another tool on the Google Container Tools on Github, that allows us to build an container image from a Dockerfile without Docker. Marvellous.

It has som benefits:

*Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. (no need to mount a docker socket anymore)

Provides a handy docker image to use in pipelines (a very light weight one, so your jobs doesn't take much time to run)
Provides a Caching system in the same repository where the images are stored, so in every job that tries to build the image, Kaniko will check first the repo cache layers and downloaded speeding up the build job.

When a build a image I check my repo and I can see the caching layers separated from the final images (see image below)

Cache Layers

Builded and Tagged Images

Deployment safety considerations

All in life has it's tradeoff, and this isn't the exception, however, is possible to joint team autonomy with security and governance, no only with this example, but in general in software development.

Some of my personal recommendations on this are:

Secrets: Always use a secret vault (in my case i use Hashicorp one)
Fine-grained permissions control with the CI/CD tunnel via impersonation and k8s agent:
- Allows you to leverage your K8s authorization capabilities to limit the permissions of what can be done with the CI/CD tunnel on your running cluster
- Lowers the risk of providing unlimited access to your K8s cluster with the CI/CD tunnel
- Segments fine-grained permissions with the CI/CD tunnel at the project or group level
- Controls permissions with the CI/CD tunnel at the username or service account
Make your high environment namespaces inmutable (on K8s namespace creation time)
Fine Grained RBAC on Gitlab Roles (I think this isn't available on Free and Self-Managed version)

Next Chapter

In the next chapter I'll wrap up all things that we cover in this first 2 releases of the series in a full functional pipeline from local to production with a small k8s cluster and i expect, that you can see all the work in action and some final remarks to allow you to test in your own projects.

Thanks for reading!

Support me

If you like what you just read and you find it valuable, then you can buy me a coffee by clicking the link in the image below, it would be appreciated.

How to build a CI/CD workflow with Skaffold for your application (Part I)

Raul Castellanos — Mon, 31 Oct 2022 20:39:26 +0000

Skaffold (part of the Google Container Tools ) was on the market since 2018, but was in 2020 when, (at least for me), they reach a prod-grade maturity level on the tool.

And I was more than fascinated on how, this tool not only can facilitate the developer work on local machines, but also, as a complete pipeline from development to production environment, if is used with a couple of other tools.

Easy and Repeatable Kubernetes Development, no matter if you are a developer, lead, platform engineer, SRE or head of Engineering, all we're agree on that 🙋🏽.

We want an easy, repeatable, reproducible development workflow, that brings more autonomy in the teams, to bring more product value to the final user in a secure way.

I want to show you, how I use Skaffold as building-block for my micro service CI/CD pipeline from local to production.

The Toolset

skaffold cli (you can use the provided docker image or install in you machine)
kubectl and Kustomize (kustomize is part of kubectl cli already)
K8s cluster (local and remote) - if you use docker-desktop you've already one cluster installed by default, to use for local development.

Isn't intended in this article to show, how to install a k8s cluster for local development, you have various alternatives like Minikube out there. Like I said, I my case since I use docker-desktop and they come with a k8s inside by default.

The Workflow

The main idea is, to use 'skaffold' as a building block from local to production environment, simplifying the tooling used by the developer and facilitating the integration with the actual gitlab repository service.

The more complex part would be, the integration and functional tests, since integration and functional tests needs the complete application and dependencies running, a little more work needs to be done to accomplish that, however isn't as complex as sounds, since I used a Kubernetes gitlab runner to run the pipelines, so, we can use the same runner to deploy the application in a special namespace, run our tests, and then, remove the application from the runner.

Be aware, that you need to do a cleanup process after each pipeline or stage run, to avoid left running process consuming capacity and space in your Kubernetes cluster and to avoid recurring in unexpected operational costs.

The Application

It's the simplest micro service you may know, but it's intended for demonstration purposes. Was made in Symfony with RoadRunner as application server, expose metrics to a prometheus metrics server, and then this metrics are fetched from the monitoring stack to be used in grafana dashboard for monitoring and observability purposes.

Now, that we have all the context clear, lets begin with the next steps: first at all, i need to setup our repository skeleton and directory structure, in order to be, as functional as possible to my intended workflow and development process.

Take into account, that this, is how I setup my repositories, and you should fit this to your expectations and operational workflow.

Additionally, my principal goal when I began to work with this GitOps approach was also: reduce the cognitive load and operational complexity for new an current colleagues, reducing also the onboarding time and the number of tools that we need to do our work.

If you need a more complex scenario for metrics scaling, i normally try to use thanos for that job, since allows me to easily scale prometheus, and get long term storage in commonly knows cloud object storage services, like Amazon S3 or GCP Cloud Storage.

Below you can find a diagram of the same application but implementing thanos.

Let's continue this guide with the simplified version of the application (the one with only prometheus and Grafana components for monitoring)

Folder Structure and Orchestration process

Lets recap on some concepts about the tooling that we're implementing here, to fulfil the promise of a, full workflow with skaffold from local to production.

I deploy on K8s cluster with kubectl and kustomize (kustomize is part of k8s bundle).
I use skaffold as workflow building block through their cli command steps (build, test, render, deploy, verify)
I build images locally with docker cli (is a pre requisite for my workflow) and in gitlab i have a couple of options alongside docker (Kaniko or Docker in Docker variations), but i'll cover that in the next steps.
I use a Makefile as command "collector" entrypoint no only for local development but for gitlab pipeline to group commands in single word ones (make run, build, unit, etc).
I use terraform declarative configuration files, to set the desired state of my working cluster (in that case the local one), in this desired state we have some pre-requisites needed in my architecture definition, like cert-manager, traefik, prometheus and grafana, like my machines on staging and production.

- deploy/
    manifests/ # the place where k8s yaml resides
         - *k8s.yaml
         - kuztomization.yaml
    overlays/ # for every environment that you want, you should have an overlay
        development/
              - *.k8s.patch.yaml
              - kuztomization.yaml
        production/
             - *.k8s.patch.yaml
             - kuztomization.yaml
   - skaffold.yaml
- infrastructure/ #terraform scripts to install cluster pre requisites vault, cert-manager, treafik
- src/ # all the source code of your application
    - Dockerfile 
- Makefile
- .gitlab-ci.yaml

Another important component of this setup is: the Dockerfile, to be able to use the same dockerfile to build images for development and production environments (with the dependencies of each of them), i build a multi-stage dockerfile that allows me to get a target for development, and a target for production, that we can point to in the skaffold build phase.

Kustomize files, (kustomization.yaml ones), allow me to declare a patch or merge of the a part of the main k8s manifest, to apply the changes that I need to change in some environment without the necessity of duplicate the entire YAML, so, for example, if I have the following k8s manifest declaring a api with 1 replica, and then, i can declare a patch to set that number to 4 replicas if the environment is production.

The following image shows you, how is a main k8s manifest and their corresponding patch for production. You can patch anything you want, adding all the data, metadata and others labels to every manifest in environment overlays.

Then, we have our skaffold file, in charge of the orchestration process of the workflow itself:

Since skaffold allows us to use kustomization as deployment strategy, i organise my profiles to do so, and also, with this, the development team has a lot of maneoveur to modify and deploy changes with zero effort.

Now, i can run everything in one shot to see how this work, so, it's everything is ok, i'll capable to access all the tools (via browser) and the API requesting them.

To run skaffold you need to run the following command:skaffold dev -p development, but since we use a Makefile as command entrypoint, you can see above that make run do the same job as we need to run skaffold in development mode.

I use my domain and a self signed certificate to access, all applications via a FQDN over https (using cert-manager and traefik for that), now, I'll be able to access all of them via that URLS (on the local machine this urls points to the loopback 127.0.0.1 in the /etc/hosts file)

We should have at lest this applications:

Grafana
Traefik Dashboard
API /Application

Lets see in this animated GIF, those applications running:

😅 with this , i already have a full local development cycle for my local environment, now the next milestone is, make my gitlab pipelines compliance with this pipeline and make the way to Low and Prod environments .

Let's stop here for now. I'll prepare the material for the next blog entry.

Next Chapter

In the next chapter of this tutorial, I'll try to implement this local workflow in gitlab pipeline, allowing me to use the tests, build, render, deploy and verify skaffold stages in my entire pipelines and deploy the application to a k8s cluster in GCP in a full GitOps manner.

Thanks for reading and see you the next week for more! 😃

A Big KUDOS to the team #skaffold for the great job, if you wan to know more about you can reach them at slack or in their repo

Support me

If you like what you just read and you find it valuable, then you can buy me a coffee by clicking the link in the image below, it would be appreciated.

Forem: Raul Castellanos

Managing database migrations safely in high replicated k8s deployment.

The Problem

When to run migrations (the workflow)

Job, InitContainer and RollingUpdates

Safety recommendations

Support me

How to build a CI/CD workflow with Skaffold for your application (Part III)

Let's recap: The Workflow

Gitlab K8s agent and Security

Deployment and Safety Recommendations for K8s Agents

Provisioning cluster with terraform

Next

Support me

How to build a CI/CD workflow with Skaffold for your application (Part II)

Lets recap the Workflow

The Makefile

The skaffold file

The .gitlab-ci.yml Pipeline

Feature Branches Workflow (trigger by merge_request commit):

Main Branch Workflow (trigger by a TAG):

BONUS: Kaniko image Builder

Deployment safety considerations

Next Chapter

Support me

How to build a CI/CD workflow with Skaffold for your application (Part I)

The Toolset

The Workflow

The Application

Folder Structure and Orchestration process

Next Chapter

Support me

When to run migrations `(the workflow)`

`Job`, `InitContainer` and `RollingUpdates`

Let's recap: `The Workflow`

Gitlab `K8s agent` and Security

Deployment and Safety Recommendations for `K8s Agents`

Provisioning cluster with `terraform`

Lets recap the `Workflow`

The `skaffold` file

The `.gitlab-ci.yml` Pipeline

BONUS: `Kaniko` image Builder