Forem: Chabane R.

Mayday, mayday! I need a scalable infrastructure to migrate on Scaleway Elements! Part 2 - Ops & Container migration

Chabane R. — Fri, 12 Nov 2021 12:55:25 +0000

Hello again!

We saw in the part 1 how to build a scalable Scaleway Elements organization, implementing a networking topology, defining a disaster recovery and centralizing IAM and key management.

In this part 2, we will talk about DevOps, monitoring, logging and migration of business applications in Kubernetes as an example.

DevOps

The Cloud is ruled by DevOps – this fact has been proven time and again. [1]

I highly recommend all my customers to adopt a DevOps culture and practices from the beginning.

Using a CI/CD tool becomes vital when you have multiples environments and a complex infrastructure configuration that depends on infrastructure as code.

Many DevOps tools exist in the market. Let's take Gitlab as an example.

Gitlab

With Gitlab, you could manage your own instance in your infrastructure or use the SaaS solution, gitlab.com.

In the SaaS solution, you can choose between:

shared runners that are managed by GitLab infrastructure and limited in minutes (Depending on your licence subscription)
specific or group runners that will be managed by your infrastructure with no limit of minutes.

If you choose the shared runners to deploy your resources in Scaleway Elements, you will need to create and save service accounts keys as variables in the Gitlab CI configuration. You will face all the security issues related to storing credentials outside of your cloud infrastructure: Key rotation, age, destruction, location, etc.

For production use, I recommend customers to use specific or group runners with jobs running in a Kubernetes Kapsule cluster.

The runner configuration that I had used for customers:

Specific or group runners only,
Deployed using Helm Chart, more easy to upgrade,
Accessibles by tags,
Locked to current project(s),
Ran only on protected branches (Production),
Assigned a specific Kubernetes Service Account,
Jobs run in separate nodepools per environment.

The Kapsule cluster that hosts those runners has its own private network which is isolated from the internet with an explicit allow security group rule for gitlab ingress traffic on port 443.

If your organization has multiple teams, you can grant specific teams access to only the runners that they use:

GitOps

GitOps builds on DevOps with Git as a single source of truth for declarative infrastructure like Kubernetes. [2]

If you have the possibility to migrate your existing workloads to Kubernetes for example, you can take advantage of GitOps practices.

GitOps Tools

Many tools exist in the market like ArgoCD and FluxCD. Personally I consider ArgoCD as the most complete tool for GitOps but you will need to manage it by yourself in your infrastructure.

ArgoCD allows you to:

Manage configurations for multiple environments using a Git repository as a single source of truth.
Review changes before deployment through pull requests on GitHub or merge requests on Gitlab.
Test and promote changes across different environments.
Roll back changes quickly.
See your applications version and status on the ArgoCD Console.

GitOps Integration

The common patterns used to integrate GitOps practices is creating multiples repositories instead of a single source.

Image repository dedicated for the business application. The dockerfile(s) will reside in this repository.
Env repository dedicated for the deployment in Kubernetes. The kubernetes manifests will reside in this repository.
Infra repository dedicated for the deployment in Scaleway Elements. The terraform plans will reside in this repository.

Let's illustrate that with a diagram:

1 - We start by deploying the Kapsule cluster using terraform and infra repo:

The gitlab runner job has a Kubernetes Service Account (KSA) which is bound with an API Key (stored in Vault) with appropriate permissions in the Env project.

2 - Build a new docker image after each git tag:

The new docker image is built and published in a centralized container registry.

Share the specific runner registered in infra repo with the image repo.

3 - Edit the docker image version of Kuberentes manifests using Kustomize.

The kubernetes workloads are updated automatically with the new docker image version using a GitOps tool like ArgoCD.

Share the specific runner registered in infra repo with the env repo and lock the runner to the current Gitlab projects.

4 - Authorize the kubernetes cluster of the env project to access docker images from the DevOps project:

We can easily develop a generator that create for each new env project a blueprint of gitlab repositories and pipelines. I had the chance to build a generator like this for a customer using Yeoman. It was a really cool challenge.

Final architecture:

Operations

Monitoring

If you have multiple Kubernetes clusters to operate, you have the possibility to centralize all of your metrics in a centralized monitoring dashboard for all of your environments (development, integration, and production).

You can achieve this centralization using:

Prometheus to collect metrics.
Thanos alongside Prometheus. It will be used for long term storing capabilities and federation.
Alert Manager to send alerts based on metrics query.
Grafana fancy dashboards.

For more details, you can consult this article: Multi-Cluster Monitoring with Thanos

Logging

Hot

When you deploy containers to your development cluster, you will need quick access to logging to analyze container errors. A good solution is to deploy Rancher in your DevOps cluster. Rancher has many features:

An admin console,
Give you a cluster overview,
Access to workloads logs and states,
Role-based Access Control, etc.

Cold

When you operate your production cluster, in the event of a disaster or failure, you will need to access the log history. The ELK might be a good solution for storing logs for the long term:

Business applications

Let's say we have docker images hosted on premise and we want to deploy them in a Scaleway Elements Service.

There are two ways to achieve that:

Lift & Shift in Virtual instance, if we had a self-managed docker platform.
Improve & Move in Serverless Containers or Kubernetes Kapsule.

Unless there is a strong dependency between the docker runtime and the business applications, there is no value to go to the Cloud if you don't use a managed services for Docker.

Serverless Containers is currently ideal if your application is internet facing and does not depend on network or security restrictions.

Kapsule is still the most used to deploy docker images, as it's fully managed by Scaleway Elements and highly maintained and secured by the cloud provider.

Configuring a Kapsule cluster

I apply the following configuration to secure a Kapsule cluster:

Stable version as possible. Do not use an upstream release.
Kapsule attached to a private network (Possible soon),
Backup of the cluster using Velero,
Pools with auto scaling & auto healing enabled,
Network Policy enabled to manage network traffic,
HTTPs Load balancer (LB) enabled for internet facing applications + LB ACLs when necessary,
Keycloak for internal applications.

Kubernetes architecture

To migrate a 3-tiers architecture in Kapsule, I commonly deploy this architecture in Kubernetes:

Depending on the use case, you can deploy additional tools like Istio.

I wrote a complete article on how to secure connectivity between a Kapsule container and RDB databases.

Docker images

When you migrate existing docker images in the cloud, it's not easy to optimize the existing layers for fear of breaking something.

What I recommend for all existing images is to:

Properly tag images (semantic versioning and/or Git commit bash),
Use Kaniko to build and push images,
Use Container Registry to store the images.

For new images:

Package a single app per container,
Build the smallest image possible,
Optimize for the Docker build cache,
Remove unnecessary tools,
Carefully consider whether to use a public image,
Use Clair for vulnerability scanning.

Cost Savings tips

When you move existing workloads in a Cloud Provider, you can save money in two differents ways:

Cost-optimization: Running the workloads in the right service with the right configuration.
Cost-cutting: Removing deprecated and unused resources.

To apply a cost-cutting, you can develop a Serverless Function that:

Stops all resources that are not Serverless on non production environments like VMs and RDB instances,
Resizes Kapsule nodepools to zero on non production environments,
Removes old docker images.

For new workloads, the best way to save money is to use Scaleway Serverless services as possible.

Conclusion

It took us around 20 days the first time we deployed such architectures. However, after applying these principles, the ensuing implementations took just few days to deploy them.

There are other subjects I would have liked to give feedback on: Data Analytics and AI implementation. We can keep it for a part 3.

In the meantime, if you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope it helped you to see how automating everything via CI/CD and using infrastructure as code anywhere will ensure you to have a scalable strategy to migrate on Scaleway Elements whatever is your business.

By the way, do not hesitate to share with peers 😊

Documentation:

[1] https://www.idexcel.com/blog/true-business-efficiency-combines-the-power-of-cloud-computing-and-devops-practices/
[2] https://www.weave.works/blog/gitops-modern-best-practices-for-high-velocity-application-development

Mayday, mayday! I need a scalable infrastructure to migrate on Scaleway Elements! Part 1 - Networking & Security

Chabane R. — Fri, 12 Nov 2021 10:14:11 +0000

That's it! We are going to migrate our on-premise applications to Scaleway Elements!

We start by deploying a first Web application to please the business team, we set up auto scaling and backups to satisfy the ops team, we encrypt the storage and the database with a key and a firewall to reassure the security team, all at a lower price.

Everything has been thought of and the migration of the first application is a real success and the business accepts to migrate other applications ... But wait ... no long-term strategy has been established to facilitate the migration of new applications 😱

Moving to the Public Cloud represents a major change within a company. A phase of adoption and acculturation is necessary for a successful transition.

A lack of a long-term vision can be costly during migrations to the Cloud and in particular to Scaleway Elements (SCW).

Organization of projects
Network topology
Centralization of security and monitoring
The DevOps platform
A plan for every migration strategy

Must be defined in advance, at the onboarding phases!

Did you also find it difficult to go further in your migration? Or are you thinking of migrating to Scaleway Elements?

Before discovering how to plan a migration in Scaleway Elements, let's see the common traps.

Traps to avoid

Poc to production

Some companies starting by developing a proof of concept put that POC into production.

A POC is a disposable, it should be dropped after the concept has been proved.

So if during the POC you created a SCW organization, used a CI/CD tool, deployed the resources using an infrastructure as code, it's not a POC but an MVP (Minimum Valuable Product) which it's different !

I saw many companies deployed their POC to production and asked a partner to perform a complete refactoring of their network topology, reorganazing the cloud projects, implementing a CI/CD and infrastructure as code and it finally costs more than they thought to save.

On Premise mindset

Some companies moving to the cloud continue to follow their on-premises practices.

Only the "business" workload should be moved to the cloud and it's called "lift & shift". All existing applications for security, networking, monitoring, etc. should be replaced either by the equivalent in the cloud provider or the cloud version of those applications.

Also for this case I saw that many companies wanted to move their IT software to the cloud which is not able to interact with cloud services.

Avoiding vendor lock-in at extreme level

Some companies moving to the cloud do not use managed services when it's possible.

Avoiding vendor lock-in has a sense when you are looking for portability of your business application and you don't want to depend on cloud SDKs. But all non-business applications for databases and storage should be hosted in managed services.

There are many other 'don't do' that can be listed in a dedicated post.

So before any move to the cloud, a strong sponsorship is needed to adopt a cloud mindset in the organization.

Choose a scalable strategy

When a customer asks to migrate to the cloud, I always recommend the deciders to think about a long-term vision and value with the selected cloud provider.

Why is the current approach and computing environment insufficient?
What are the primary metrics that you want to optimize by using the public cloud?
How long do you plan to use this cloud provider? Do you consider this solution permanent?

When the vision is defined, the strategy to adopt is clearer:

Identifying candidate workloads
Identifying applicable patterns
Identifying candidate topologies
Prioritizing workloads
Select initial workload to put in the public cloud
Setting the Scaleway Elements organization, projects, and policies
Implementing the network topology
Defining the disaster recovery plan
Setting the DevOps platform
Start workloads migration

Organization

When you have created the Scaleway Elements organization for your company, the first step to do is defining the projects roles. What I recommend to customers is to separate the business workloads from the operational workloads.

These roles help:

to apply granular permissions
to facilitate infrastructure as code automation
to centralize IT operations

Network

Many network topologies types exist and help to ensure communications between network node.

As we have a connection to establish between differents projects, we need to ensure a secure connectivity between networks.

1 - Create a private network for each project

2 - Create a public gateway with a reserved IP for each project and attach it to the private network.

3 - For each Kubernetes Kapsule cluster of each environment project, create an HTTPs Load balancer (LB) and whitelist the Public Gateway IPs of the Security and DevOps projects using LB ACLs.

At the time of writing this article, a Kubernetes Kapsule cluster cannot be attached to the private network and the public gateway. It's coming soon.

SecOps

Identity and Access Management

When you have a LDAP in your organization, it is common to connect it to your internal tools. There are many solutions to help you set up identity and access management, one of the most tool is Keycloak.

Keycloak supports multiples procotols:

OpenID Connect
OAuth 2.0
SAML 2.0

A support for Single Sign-On and Single Sign-Out, Admin Console, etc.

The tool could be centralized in a dedicated project and used to centralize authentication on each application in your organization. The following diagram illustrates IAM management:

Key management

There are three very sensitives resources in SCW and in any cloud provider that we need to protect whatever the price:

API Keys
Cryptographic keys
Secrets

Virtual instance images could also be critical for some organizations.

API Keys

There are some important best practices to secure the sensitives credentials:

Create single-purpose API keys
Don’t embed API keys in code
For easier visibility and auditing, central store API keys in a solution like Vault and in a dedicated project.

I wrote a complete tutorial on Securing access to Scaleway Elements API Keys from Gitlab CI

Cryptographic keys

If you need to encrypt your data using a cryptopgraphic key, there are some important best practices to apply:

Hosting encryption keys in a separate project,
For critical projects, some enterprises save the keys in a separate organization,
Least privilege and separation of duties.

Secrets

In any IT project, you will need two types of credentials:

Secrets for running your business applications like DB credentials.
Secrets to access shared applications like Elasticsearch, ArgoCD, Git repositories, etc.

In Scaleway Elements, you can host shared secrets in the Security project and keep secrets used by business applications in a Kubernetes secret.

There are some important best practices to manage secrets:

Use the replication feature when creating secrets in Vault.
Reference secrets by their version number rather than using the latest alias.
Disable secret versions before destroying them or deleting secrets.

The following diagram illustrates the key management:

Disaster recovery

A Disaster Recovery (DR) involves a set of policies, tools and procedures to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster.

In Scaleway Elements, there are managed services to help you implement a disaster recovery plan, such as backup services on your databases and virtual instances. But in the event of a zone/region failure or human error that deleted a project, you will need to react very quickly to restore your data and applications.

Let's take an example of a kubernetes cluster and a RDB database that we want to backup and restore in the event of an outage:

Backup

Velero is an open source disaster recovery tool used to backup an entire Kubernetes cluster and save the backup in a bucket. A Scaleway Serverless function could be used to download and upload backups periodecally to a bucket:

Restore after a zone failure

In the event of a zone failure, a CI/CD pipeline could be run to restore the Kubernetes cluster with Velero and restore the databases using Scaleway CLI.

In the case of a re-creation of the RDB instance, you will also need to update the host and port information.

Restore after a region failure

In the event of a region failure or a project being deleted, CI/CD pipelines could be run to recreate the Kubernetes cluster and the RDB instances with Terraform, restore the Kubernetes cluster with Velero and restore the database using Scaleway CLI.

Conclusion

In this part we saw how it easy to build a scalable SCW organization, implementing a network topology, defining a disaster recovery and centralizing the IAM and key management.

What's next ?

In the second part we will see how we can deploy a DevOps platform using Gitlab and following the GitOps practices. We will also see how to centralize monitoirng and logging using Open Source tools. We will finish with an example of a migration of Docker applications in Kubernetes Kapsule and finishing with cost saving tips.

Securing the connectivity between a Scaleway Kubernetes Kapsule application and Scaleway RDB database

Chabane R. — Fri, 05 Nov 2021 08:22:46 +0000

It is very easy today to establish a connection between a container in Kubernetes and a relational database server, just create a SQL user and open a TCP connection. In cloud computing, in the case of Scaleway Elements, the equivalent is connecting a container in a Scaleway Kubernetes Kapsule cluster to a Scaleway Relational Database instance (RDB).

Important points should be taken into account in setting up this connectivity.

Which network topology to choose? How to authenticate and authorize the connection to the RDB instance? Can I publicly expose a RDB instance?

Which architecture could be the most efficient, maintainable and scalable?

Scenarios

RDB service supports the following scenarios for accessing the RDB instance:

A virtual instance in the same project
A virtual instance in a different project
A client application through the internet
The private network scenario is actually not possible. A feature request is opened and it's ongoing.

The scenarios that concern us are the first two:

A Kapsule cluster and a RDB instance in the same project
A Kapsule cluster and a RDB instance in a different project

Let's discover the possible architectures that could be used to implement each scenario.

Direct communication with public IP and authorized networks

In this architecture, our RDB instance is isolated on Scaleway network and accessible through public IP address to only Kapsule cluster that requires access to it. Pods have access to RDB database using Username/Password.

Public Gateway is not available yet for Kubernetes Kapsule and RDB instance, see featurer request, so you need to whitelist all the Kubernetes nodes IPs.

Direct communication in separate projects

In this architecture, our RDB instance is isolated on its own project accessible through public IP address to only Kapsule cluster that requires access to it. Pods have access to RDB database using Username/Password.

Each architecture has its own advantages and disadvantages but all apply project isolation best practices for securing sensitive data in RDB.

Let's implement the scenario 1.

Prerequisites

Download, install, and configure the Scaleway CLI.
Terraform
Kubectl
Kustomize

Architecture

The overall architecture that we will implement during this article is as follows:

Objectives

During this section of the workshop:

With Terraform

We will create a Kapsule cluster
a higly available RDB instance
a virtual instance to act as a bastion to access RDB from outside
a security group for the virtual instance

With Kubectl

We will deploy a metabase application
A load balancer with Treafik 2
a SSL certificate

The article is divided into four sections:

Configuring the bastion
Creating a Kubernetes Kapsule cluster using Terraform
Securing sensitive data in Scaleway RDB
Securing the connectivity between a Kubernetes Kapsule application and a RDB database

Configuring the bastion

In this section we will deploy the following SCW resources:

a virtual instance to act as a bastion to access RDB from outside
a security group to allow only authorized users to access the virtual instance

Security Groups

Security Groups allows us to restrict the inbound and outbound network traffic to and from a virtual instance. In our case, we implement the following rule:

A rule to restrict access to the virtual instance from the SSH port to only authorized networks.

Create a terraform file infra/plan/sg.tf

resource "scaleway_instance_security_group" "bastion" {
  name                    = "bastion"
  inbound_default_policy  = "drop"
  outbound_default_policy = "accept"

 dynamic "inbound_rule" {
    for_each = var.authorized_source_ranges
    content {
        action = "accept"
        port   = "22"
        ip     = inbound_rule.value
    }
  }
}

Bastion

In order to access the RDB instance from outside, we need to create a bastion host. We can achieve that by creating a virtual instance. During the initialization of the instance we install the PostgreSQL client.

We also create a SSH key to connect to the virtual instance. You can remove this resource if you want to manage the keys outside of terraform.

Create a terraform file infra/plan/bastion.tf

resource "scaleway_instance_ip" "bastion" {}

resource "scaleway_instance_server" "bastion" {
  depends_on = [scaleway_account_ssh_key.bastion]

  name              = "bastion"
  type              = var.bastion_instance_type
  image             = "ubuntu_xenial"

  security_group_id = scaleway_instance_security_group.bastion.id

  ip_id = scaleway_instance_ip.bastion.id

  tags = ["bastion", var.env]

  user_data = {
    env        = var.env
    cloud-init = file("${path.module}/cloud-init.sh")
  }
}

resource "scaleway_account_ssh_key" "bastion" {
    name       = "bastion"
    public_key = var.bastion_public_ssh_key
}

Create a script file cloud-init.sh

#!/bin/sh
apt update

apt install -y unzip

# install awscli
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# install psql
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
apt update
apt install -y postgresql-client-13

Let's configure our terraform.

Create a terraform file infra/plan/variable.tf

variable "zone" {
  type = string
}

variable "region" {
  type = string
}

variable "env" {
  type = string
}

variable "authorized_source_ranges" {
  type        = list(string)
  description = "Addresses or CIDR blocks which are allowed to connect to Virtual Instance."
}

variable "bastion_instance_type" {
  type = string
}

variable "bastion_public_ssh_key" {
  type = string
}

Add a infra/plan/version.tf file

terraform {
  required_providers {
    scaleway = {
      source  = "scaleway/scaleway"
      version = "2.1.0"
    }
  }
  required_version = ">= 0.13"
}

Add a infra/plan/provider.tf file

provider "scaleway" {
  zone            = var.zone
  region          = var.region
}

Add a infra/plan/backend.tf

terraform {
  backend "s3" {
  }
}

And a ìnfra/plan/output.tf

output "bastion_ip" {
  value = scaleway_instance_ip.bastion.address
}

Now, export the following variables and create a bucket to save your terraform states.

cat <<EOF >~/.aws/credentials
[default]
aws_access_key_id=<SCW_ACCESS_KEY>
aws_secret_access_key=<SCW_SECRET_KEY>
region=fr-par
EOF

export SCW_ACCESS_KEY=<SCW_ACCESS_KEY>
export SCW_SECRET_KEY=<SCW_SECRET_KEY>
export SCW_REGION=fr-par

sed -i "s/<SCW_ACCESS_KEY>/$SCW_ACCESS_KEY/g; s/<SCW_SECRET_KEY>/${SCW_SECRET_KEY}/g;" ~/.aws/credentials 

export ENV=dev
aws s3api create-bucket --bucket company-$ENV-terraform-backend --endpoint-url https://s3.$SCW_REGION.scw.cloud
aws s3api put-bucket-versioning --bucket company-$ENV-terraform-backend --versioning-configuration Status=Enabled --endpoint-url https://s3.$SCW_REGION.scw.cloud

Create a infra/plan/config/dev/terraform.tfvars:

zone                           = "fr-par-1"
region                         = "fr-par"
env                            = "dev"

authorized_source_ranges       = ["<AUTHORIZED_NETWORK>"]


bastion_instance_type          = "STARDUST1-S"
bastion_public_ssh_key         = "<BASTION_PUBLIC_SSH_KEY>"

Create a infra/plan/config/dev/s3.backend and deploy the infrastructure:

bucket                      = "company-dev-terraform-backend"
key                         = "terraform.tfstate"
region                      = "fr-par"
endpoint                    = "https://s3.fr-par.scw.cloud"
skip_credentials_validation = true
skip_region_validation      = true

cd infra/plan

sed -i "s,<AUTHORIZED_NETWORK>, $(curl -s http://checkip.amazonaws.com/),g; s,<BASTION_PUBLIC_SSH_KEY>,$(cat <PATH_TO_SSH_PUB>/id_rsa.pub),g;" terraform.tfvars

terraform init --backend-config=config/$ENV/s3.backend -reconfigure

terraform validate

terraform apply -var-file=config/$ENV/terraform.tfvars

Let's check if all the resources have been created and are working correctly

Reserved IP

Security Groups

Virtual instance

Let's check the connection

ssh -i <PRIVATE_KEY_PATH> root@$(terraform output --raw bastion_ip)

Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-159-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Nov  4 21:13:14 UTC 2021

  System load:  0.09              Processes:           83
  Usage of /:   19.4% of 8.86GB   Users logged in:     0
  Memory usage: 15%               IP address for ens2: 10.69.86.243
  Swap usage:   0%

Creating a Kubernetes Kapsule cluster using Terraform

In the section we created our network stack. In this part we will create and configure the Kubernetes Kapsule cluster.

The following resources will be created:

Kubernetes Kapsule cluster
Kubernetes Kapsule pools
Traefik2 Load balancer

Kubernetes Kapsule cluster

Our Kubernetes Kapsule cluster is hosted in the Scaleway network. Each node of a pool has its own public IP, there is no such a mecanism that permit to access the private IPs of the cluster from outside. So all of the communications between RDB and Kapsule will be throught the public internet.

Scaleway is working on a feature to attach a Kapsule cluster to a private network. See the feature request.

Create the terraform file infra/plan/kapsule.tf:

resource "scaleway_k8s_cluster" "kapsule" {
  name        = "kapsule-${var.env}"
  description = "${var.env} cluster"
  version     = var.kapsule_cluster_version
  cni         = "calico"
  tags        = [var.env]

  autoscaler_config {
    disable_scale_down              = false
    scale_down_delay_after_add      = "5m"
    estimator                       = "binpacking"
    expander                        = "random"
    ignore_daemonsets_utilization   = true
    balance_similar_node_groups     = true
    expendable_pods_priority_cutoff = -5
  }

  auto_upgrade {
    enable                        = true
    maintenance_window_start_hour = 4
    maintenance_window_day        = "sunday"
  }
}

resource "scaleway_k8s_pool" "default" {
  cluster_id  = scaleway_k8s_cluster.kapsule.id
  name        = "default"
  node_type   = var.kapsule_pool_node_type
  size        = var.kapsule_pool_size
  autoscaling = true
  autohealing = true
  min_size    = var.kapsule_pool_min_size
  max_size    = var.kapsule_pool_max_size
}

Complete the file infra/plan/variable.tf:

variable "kapsule_cluster_version" {
  type = string
}

variable "kapsule_pool_size" {
  type = number
}

variable "kapsule_pool_min_size" {
  type = number
}

variable "kapsule_pool_max_size" {
  type = number
}

variable "kapsule_pool_node_type" {
  type = string 
}

Complete the file infra/plan/config/$ENV/terraform.tfvars:

kapsule_cluster_version        = "1.22"
kapsule_pool_size              = 2
kapsule_pool_min_size          = 2
kapsule_pool_max_size          = 4
kapsule_pool_node_type         = "gp1-xs"

Let's deploy our cluster

cd infra/plan

terraform apply -var-file=config/$ENV/terraform.tfvars

Let's check if the cluster and the pools have been created and are working correctly:

Kubernetes Kapsule Cluster

Kubernetes Kapsule Pool

Kubernetes Kapsule NodePools

Enable the Traefik load balancer using the Scaleway CLI:

scw k8s cluster update $(scw k8s cluster list | grep kapsule-${ENV} | awk '{ print $1 }') ingress=traefik2 region=$SCW_REGION

Let's check if Traefik 2 has been enabled

Get the Kube config file and test the cluster access:

mkdir -p ~/.kube/$ENV
scw k8s kubeconfig get $(scw k8s cluster list | grep kapsule-${ENV} | awk '{ print $1 }') region=$SCW_REGION > ~/.kube/$ENV/config
export KUBECONFIG=~/.kube/$ENV/config

kubectl get nodes 

NAME                                             STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP     OS-IMAGE                        KERNEL-VERSION     CONTAINER-RUNTIME
scw-kapsule-dev-default-ad56eb1786504a99957ea8   Ready    <none>   100m   v1.22.3   10.66.32.141   212.47.252.20   Ubuntu 20.04.1 LTS fc08d0ff0a   5.4.0-80-generic   containerd://1.5.5
scw-kapsule-dev-default-c4056d33bede4f3883fade   Ready    <none>   100m   v1.22.3   10.66.242.89   51.15.205.158   Ubuntu 20.04.1 LTS fc08d0ff0a   5.4.0-80-generic   containerd://1.5.5

To expose Traefik 2 with a Scaleway LoadBalancer, create the file infra/k8s/traefik-loadbalancer.yml:

apiVersion: v1
kind: Service
metadata:
  name: traefik-ingress
  namespace: kube-system
  labels:
    k8s.scw.cloud/ingress: traefik2
spec:
  type: LoadBalancer
  ports:
    - port: 80
      name: http
      targetPort: 8000
    - port: 443
      name: https
      targetPort: 8443
  selector:
    app.kubernetes.io/name: traefik

Deploy the configuration:

kubectl create -f infra/k8s/traefik-loadbalancer.yml

Verify that the LoadBalancer has been deployed correctly:

kubectl -n kube-system get svc traefik-ingress

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
traefik-ingress   LoadBalancer   10.43.99.132   51.159.74.228   80:31912/TCP,443:30860/TCP   16s

See Exposing services in Scaleway Kubernetes for more details.

To avoid losing the IP, let's reserve this one:

export TRAEFIK_EXTERNAL_IP=$(kubectl get svc traefik-ingress -n kube-system -o json | jq -r .status.loadBalancer.ingress[0].ip)
kubectl patch svc traefik-ingress -n kube-system --type merge --patch "{\"spec\":{\"loadBalancerIP\": \"$TRAEFIK_EXTERNAL_IP\"}}"

Securing sensitive data in Scaleway RDB

Our Kapsule Kubernetes cluster is now active. In this section we will configure the RDB Instance.

The following resources will be created:

A highly available RDB Instance
A database and a user for metabase
ACLs to restrict the traffic to only Kubernetes Kapsule pools nodes and Bastion public IP

RDB instance

The RDB Instance used is a PostgreSQL database server
The Multiples zones option is enabled to ensure high-availability
Automated backup is enabled
We create a database and a user for later

Create a terraform file infra/plan/rdb.tf

resource "random_string" "db_name_suffix" {
  length  = 4
  special = false
  upper   = false
}

resource "scaleway_rdb_instance" "rdb" {
  name              = "postgresql-${var.env}"
  node_type         = var.rdb_instance_node_type
  volume_type       = var.rdb_instance_volume_type
  engine            = var.rdb_instance_engine
  is_ha_cluster     = var.rdb_is_ha_cluster
  disable_backup    = var.rdb_disable_backup
  volume_size_in_gb = var.rdb_instance_volume_size_in_gb
  user_name         = "root"
  password          = var.rdb_user_root_password
}

resource "scaleway_rdb_database" "metabase" {
  instance_id    = scaleway_rdb_instance.rdb.id
  name           = "metabase"
}

resource "scaleway_rdb_user" "metabase" {
  instance_id = scaleway_rdb_instance.rdb.id
  name        = "metabase"
  password    = var.rdb_user_metabase_password
  is_admin    = false
}

Complete the file infra/plan/variable.tf:

variable "rdb_is_ha_cluster" {
  type = bool
}

variable "rdb_disable_backup" {
  type = bool
}

variable "rdb_instance_node_type" {
  type = string
}

variable "rdb_instance_engine" {
  type = string
}

variable "rdb_instance_volume_size_in_gb" {
  type = string
}

variable "rdb_user_root_password" {
  type = string
}

variable "rdb_user_metabase_password" {
  type = string
}

variable "rdb_instance_volume_type" {
  type = string
}

RDB ACLs

Each node in the Kubernetes Kapsule has an attached ephemeral IP. For now, we cannot reserve the node IP or use Public Gateway with Kubernetes Kapsule. If we put the actual node IPs manually, the IP can change if an autoscaling occurs. A temporary solution is to create a Kubernetes cronjob that refresh the RDB ACLs each minute with the current node IPs.

Complete the file infra/plan/rdb.tf with the following resources:

resource "scaleway_rdb_acl" "rdb" {
  instance_id = scaleway_rdb_instance.rdb.id

  dynamic "acl_rules" {
    for_each = concat(var.rdb_acl_rules, [{
        ip          =  "${scaleway_instance_ip.bastion.address}/32"
        description = "Bastion IP"
    }])
    content {
        ip          = acl_rules.value["ip"]
        description = acl_rules.value["description"]
    }
  }
}

Complete the ìnfra/plan/output.tf

output "rdb_endpoint_ip" {
  value = scaleway_rdb_instance.rdb.endpoint_ip
}

output "rdb_endpoint_port" {
  value = scaleway_rdb_instance.rdb.endpoint_port
}

Complete the file infra/plan/variable.tf:

variable "rdb_acl_rules" {
  type = list(object({
      ip          = string
      description = string
  }))
}

Complete the file infra/plan/config/$ENV/terraform.tfvars:

rdb_instance_node_type         = "db-gp-xs"
rdb_instance_engine            = "PostgreSQL-13"
rdb_is_ha_cluster              = true
rdb_disable_backup             = false
rdb_instance_volume_type       = "bssd"
rdb_instance_volume_size_in_gb = "50"
rdb_user_root_password         = "<RDB_ROOT_USER_PWD>"
rdb_user_metabase_password     = "<RDB_METABASE_USER_PWD>"
rdb_acl_rules                  = [{
    ip          = "<KAPSULE_NODEPOOL_IP_1>"
    description = "Kapsule dev node 1"
  },
  {
    ip          = "<KAPSULE_NODEPOOL_IP_2>"
    description = "Kapsule dev node 2"
  }]

Let's deploy our RDB instance

cd infra/plan

KAPSULE_NODEPOOL_IP_1=$(scw k8s node list cluster-id=$(scw k8s cluster list | grep kapsule-${ENV} | awk '{ print $1 }') | awk '{ print $4 }' | sed -n '2 p')
KAPSULE_NODEPOOL_IP_2=$(scw k8s node list cluster-id=$(scw k8s cluster list | grep kapsule-${ENV} | awk '{ print $1 }') | awk '{ print $4 }' | sed -n '3 p')

sed -i "s/<RDB_ROOT_USER_PWD>/$RDB_ROOT_USER_PWD/g; s/<RDB_METABASE_USER_PWD>/${RDB_METABASE_USER_PWD}/g;  s,<KAPSULE_NODEPOOL_IP_1>,${KAPSULE_NODEPOOL_IP_1}/32,g; s,<KAPSULE_NODEPOOL_IP_2>,${KAPSULE_NODEPOOL_IP_2}/32,g; " config/$ENV/terraform.tfvars

terraform apply -var-file=config/$ENV/terraform.tfvars

The secret should be stored in a secret storage like Vault. You can deploy a vault in your Kapsule or in a virtual instance and retrieve the secret in terraform using the vault provider. This method avoid to store the password in the terraform state.

Add the connect privilege to the metabase RDB user:

scw rdb privilege set instance-id=$(scw rdb instance list | grep postgresql-${ENV} | awk '{ print $1 }') database-name=metabase user-name=metabase permission=all

Let's check if all the resources have been created and are working correctly:

RDB HA instance

RDB backup enabled

RDB databases

RDB Users

RDB ACLs

Let's check the connection

ssh -i <PRIVATE_KEY_PATH> root@$(terraform output --raw bastion_ip)

Run the psql command:

psql -h <RDB_ENDPOINT_IP> -p <RDB_ENDPOINT_PORT> -U metabase -d metabase

psql (13.3 (Ubuntu 13.3-1.pgdg16.04+1))
SSL connection (protocol: TLSv1.2, cipher: <CIPHER>, bits: 256, compression: off)
Type "help" for help.

metabase=>

Securing the connectivity between a Kubernetes Kapsule application and a RDB database

Our RDB instance is now available. In this section, we put them all together and deploy Metabase to Kubernetes and connect it to the RDB database. Our objectives are to:

Deploy the metabase application.
Create a DNS zone and a DNS record for the metabase application.
Create the SSL certificates

Kustomize files

Let's create the Kustomize base files.

Create the metabase deployment infra/k8s/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metabase
  labels:
    app: metabase
spec:
  selector:
    matchLabels:
      app: metabase
  replicas: 1
  template:
    metadata:
      labels:
        app: metabase
    spec:
      containers:
        - name: metabase
          image: metabase/metabase
          imagePullPolicy: IfNotPresent

Create the metabase service infra/k8s/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: metabase
  labels:
    app: metabase
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  selector:
    app: metabase

Create a secret to save the RDB user password infra/k8s/base/database-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: metabase
type: Opaque
data:
  password: metabase

Create the ingress file infra/k8s/base/ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: metabase
spec:
  rules:
  - host: metabase.<DOMAIN_NAME>
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: metabase
            port:
              number: 8000

Replace by your domain name.

Create the kustomization file infra/k8s/base/kustomization.yaml:

namespace: metabase

resources:
- deployment.yaml
- service.yaml
- database-secret.yaml
- ingress.yaml

Now let's create the files for the dev environment:

infra/k8s/envs/dev/database-secret.patch.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: metabase
type: Opaque
data:
  password: <MB_DB_PASS>

infra/k8s/envs/dev/deployment.patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metabase
  labels:
    app: metabase
spec:
  selector:
    matchLabels:
      app: metabase
  replicas: 1
  template:
    metadata:
      labels:
        app: metabase
    spec:
      containers:
        - name: metabase
          image: metabase/metabase
          imagePullPolicy: IfNotPresent
          env:
          - name: MB_DB_TYPE
            value: postgres
          - name: MB_DB_HOST
            value: "<MB_DB_HOST>"
          - name: MB_DB_PORT
            value: "<MB_DB_PORT>"
          - name: MB_DB_DBNAME
            value: metabase
          - name: MB_DB_USER
            value: metabase
          - name: MB_DB_PASS
            valueFrom:
              secretKeyRef:
                name: metabase
                key: password

infra/k8s/envs/dev/ingress.patch.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: metabase
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
  - host: metabase.dev.<DOMAIN_NAME>
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: metabase
            port:
              number: 80
  tls:
  - secretName: metabase-tls
    hosts:
    - metabase.dev.<DOMAIN_NAME>

Replace by your domain name.

infra/k8s/envs/dev/kustomization.yaml:

namespace: metabase

resources:
- ../../base

patchesStrategicMerge:
- database-secret.patch.yaml
- deployment.patch.yaml
- ingress.patch.yaml

Let's prepare the kubernetes files.

cd infra/k8s/envs/dev

RDB_METABASE_USER_PWD=$(echo -n $RDB_METABASE_USER_PWD | base64 -w 0 )
sed -i "s/<MB_DB_PASS>/$RDB_METABASE_USER_PWD/g" database-secret.patch.yaml

sed -i "s/<MB_DB_HOST>/$(terraform output --raw rdb_endpoint_ip)/g; s/<MB_DB_PORT>/$(terraform output --raw rdb_endpoint_port)/g;" deployment.patch.yaml

Create the DNS record

To add an external domain name to Scaleway, please follow the official documentation: How to add an external domain to DNS
.

Create the DNS zone:

scw dns zone create domain=<DOMAIN_NAME> subdomain=dev

Domain       <DOMAIN_NAME>
Subdomain    dev
Ns.0         ns0.dom.scw.cloud
Ns.1         ns1.dom.scw.cloud
NsDefault.0  ns0.dom.scw.cloud
NsDefault.1  ns1.dom.scw.cloud
Status       active
UpdatedAt    now
ProjectID    <PROJECT_ID>

Create the DNS record:

scw dns record add dev.<DOMAIN_NAME> name=metabase data=$TRAEFIK_EXTERNAL_IP type=A

DATA           NAME      PRIORITY  TTL  TYPE  COMMENT  ID
51.159.74.228  metabase  0         300  A     -        <DNS_RECORD_ID>

Create the SSL certificates

To make metabase more secure, we need to deploy Cert-manager to create Let's Encrypt TLS certificates:

More information in the documentation: Deploying Cert Manager.

Use the command below to install cert-manager and its needed CRD (Custom Resource Definitions):

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.6.0/cert-manager.yaml

customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

Create a cluster issuer that allow you to specify:

the Let’s Encrypt server, if you want to replace the production environment with the staging one.
the mail used by Let’s Encrypt to warn you about certificate expiration.

Create the file infra/k8s/cluster-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <MAILING_LIST>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: issuer-account-key
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik

kubectl create -f infra/k8s/cluster-issuer.yaml

kubectl get ClusterIssuer letsencrypt-prod

NAME               READY   AGE
letsencrypt-prod   True    32s

Now we can deploy metabase:

cd infra/k8s/envs/dev

kubectl create namespace metabase
kubectl config set-context --current --namespace=metabase
kustomize build . | kubectl apply -f -

kubectl get ingress

NAME                        CLASS     HOSTS                                  ADDRESS   PORTS     AGE
cm-acme-http-solver-n8x5s   traefik   metabase.dev.<DOMAIN_NAME>            80        1s
metabase                    traefik   metabase.dev.<DOMAIN_NAME>            80, 443   5s

kubectl get ingress

NAME       CLASS     HOSTS                                  ADDRESS   PORTS     AGE
metabase   traefik   metabase.dev.<DOMAIN_NAME>            80, 443   2m16s

kubectl get deploy metabase

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
metabase   1/1     1            1           2m

kubectl get svc metabase

NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
metabase   ClusterIP   10.39.118.166   <none>        80/TCP    2m40s

kubectl get secret metabase
NAME       TYPE     DATA   AGE
metabase   Opaque   1      7m30s

Let's check if the connection to the database has been completed:

kubectl logs deploy/metabase

[..]
2021-11-04 19:30:15,524 INFO db.setup :: Verifying postgres Database Connection ...
2021-11-04 19:30:16,029 INFO db.setup :: Successfully verified PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) application database connection. ✅
2021-11-04 19:30:16,030 INFO db.setup :: Running Database Migrations...
[..]
2021-11-04 19:30:56,208 INFO metabase.core :: Metabase Initialization COMPLETE

After a few minutes of waiting, you will see that the metabase is accessible via HTTPS!

Ok then!

Conclusion

Congratulations! You have completed this long workshop. In this series we have:

Created a highly-available Scaleway RDB instance
Configured a Scaleway Kubernetes Kapsule cluster with fine-grained access control to RDB instance
Tested the connectivity between a Kubernetes container and a RDB database.
Secured the access to the metabase application

That's it!

Clean

To clean the resoures, run the following commands:

terraform destroy -var-file=config/$ENV/terraform.tfvars

scw dns record delete dev.<DOMAIN_NAME> name=metabase data=$TRAEFIK_EXTERNAL_IP type=A

Final Words

I will update the article as soon as private network and/or public gateway are available for Kapsule and RDB.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I have helped you answer some of the hard questions about connecting Kapsule cluster to RDB instance.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

Securing access to Scaleway Elements API Keys from Gitlab CI

Chabane R. — Fri, 15 Oct 2021 11:26:30 +0000

How many api keys are stored per day as variables in the Gitlab CI configuration?

When a Scaleway Elements API Key is saved in Gitlab, we face all the security issues of storing credentials outside of the cloud infrastructure: Access, authorization, key rotation, age, destruction, location, etc.

There are 2 common reasons for developers to store GCP credentials in Gitlab CI:

They use shared runners.
They use specific/group runners deployed in a Scaleway Kubernetes Kapsule cluster but do not use (or do not know about) Gitlab additional configurations.

The alternative that Gitlab CI proposes for users is to mount a secret volume to the runner pods that are created for each build.

Prerequisites

Install the following tools:

And create 2 projects in your Scaleway organization:

devops
development

Working with Kapsule

The first step is to create the Kapsule devops cluster and configuring our environment [1].

Generate the API Key for the devops project. You can create one by following the documentation How to generate an API key.

scw init
scw k8s cluster create name=kapsule-devops

Let's create a pool for the runner jobs:

scw k8s pool create cluster-id=$(scw k8s cluster list | grep kapsule-devops | awk '{ print $1 }') name=dev node-type=GP1_XS size=2

Add a taint

kubectl taint nodes gitlab-runner-jobs-dev-reserved=true:NoSchedule --selector=k8s.scaleway.com/pool-name=dev

Configure kubectl to communicate with the cluster:

scw k8s kubeconfig install kapsule-devops

Create the namespace for the dev runner:

kubectl create namespace dev

Create the Kubernetes service account to use for specific runner:

kubectl create serviceaccount --namespace dev app-deployer

Generate an API Key for the specific runner.

Note: For easier visibility and auditing, I recommend to centrally store API keys in a dedicated project and in an external tools like Vault.

To allow the specific runner to impersonate the API Key we need to store the credentials in a Kubernetes secret.

kubectl create secret generic dev-api-key --from-file ~/.config/scw/config.yaml -n dev

Assign the API Key to the Gitlab runner

The next step is to mount the secret as a data volume [4].

Start by installing Helm:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Add Gitlab Helm package:

helm repo add gitlab https://charts.gitlab.io

Configure the runner:

Create the file values.yaml:

imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
metrics:
  enabled: true
runners:
  image: ubuntu:18.04
  config: |
   [[runners]]
     [runners.kubernetes]
        [[runners.kubernetes.volumes.secret]]
          name = "dev-api-key"
          mount_path = "/root/.config/scw"
          read_only = true
  locked: true
  pollTimeout: 360
  protected: true
  serviceAccountName: app-deployer
  privileged: false
  secret: dev-runner-tokens
  namespace: dev
  builds:
    cpuRequests: 100m
    memoryRequests: 128Mi
  services:
    cpuRequests: 100m
    memoryRequests: 128Mi
  helpers:
    cpuRequests: 100m
    memoryRequests: 128Mi
  tags: "k8s-dev-runner"
  nodeSelector: 
     k8s.scaleway.com/pool-name: dev
  nodeTolerations:
    - key: "gitlab-runner-jobs-dev-reserved"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

You can find the description of each attribute in the Gitlab runner charts repository [2]

Get the Gitlab registration token from Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section and create the following secret:

kubectl create secret generic dev-runner-tokens --from-literal=runner-token='' --from-literal=runner-registration-token='<TOKEN>' -n dev

Install the runner:

helm install -n dev app-dev-runner -f values.yaml gitlab/gitlab-runner

Using the specific runner in Gitlab CI

Create the pipeline .gitlab-ci.yml:

stages:
  - dev

infra:
  stage: dev
  image:
    name: scaleway/cli:v2.3.1
  script: 
    - /scw k8s cluster create name=kapsule-dev
    - /scw k8s pool create cluster-id=$(/scw k8s cluster list | grep kapsule-dev | awk '{ print $1 }') name=apps node-type=DEV1_M size=2
  tags:
    - k8s-dev-runner
  only:
    - main

The job will create a Kapsule cluster in the development project. We can follow the same steps for a production environment.

If you want to change the API Key, you just need to delete and recreate the dev-api-key secret.

Access the Kapsule cluster

You can follow the same steps to connect to the Kapsule Cluster from your Gitlab job.

Create a secret:

kubectl create secret generic dev-kapsule-config --from-file ~/.kube/config -n dev

and mount it as a data volume:

        [[runners.kubernetes.volumes.secret]]
          name = "dev-kapsule-config"
          mount_path = "/root/.kube"
          read_only = true

In this example, we used a kubeconfig with admin access. You should use RBAC to restrict access to only specific Kubernetes resources that the runner needs.

Implementing Gitlab Flow

Environment branches with Gitlab Flow is a branching strategy and workflow. Suppose you have additional environments like a pre-production environment and a production environment. Deploy the main branch to your development environment. To deploy to pre-production, create a merge request [3] from the main branch to the pre-prod branch. Go live by merging the pre-prod branch into the production branch.

Add the following script ./utils/autoMergeRequest.sh:

#!/bin/bash

[[ $CI_PROJECT_URL =~ ^https?://[^/]+ ]] && CI_PROJECT_URL="${BASH_REMATCH[0]}/api/v4/projects/"

BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"Deployment to ${TARGET_BRANCH}\",
    \"assignee_id\":\"${GITLAB_USER_MAILINGLIST_ID}\"
}";

LISTMR=`curl --silent "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests?state=opened" --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}"`;
COUNTBRANCHES=`echo ${LISTMR} | grep -o "\"source_branch\":\"${CI_COMMIT_REF_NAME}\"" | wc -l`;

if [ ${COUNTBRANCHES} -eq "0" ]; then
    curl -X POST "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests" \
        --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}" \
        --header "Content-Type: application/json" \
        --data "${BODY}";

    echo "Opened a new merge request: Deployment to ${TARGET_BRANCH} and assigned to you";
    exit;
fi

echo "No new merge request opened";

It's a common pattern to receive the merge request from a mailing list. Create a Gitlab user with this mailing list and add the variable CI/CD GITLAB_USER_MAILINGLIST_ID with the user ID. You can get the ID using the URL https://gitlab.com/api/v4/users?username=<USERNAME>.

Add the Variable CI/CD PRIVATE_TOKEN, you can create one from Settings > Project Access Tokens.

Example of a gitlab-ci.yml:

stages:
  - build
  - dev
  - preprod
  - prod

build:
  stage: build
  script: 
    - echo 'build'
    - TARGET_BRANCH=main ./utils/autoMergeRequest.sh
  only:
    - /^feature\/*/
    - /^hotfix\/*/

dev:
  stage: dev
  script: 
    - echo 'deploy dev'
    - TARGET_BRANCH=preprod ./utils/autoMergeRequest.sh
  tags:
    - k8s-dev-runner
  only:
    - main

preprod:
  stage: preprod
  script: 
    - echo 'deploy preprod'
    - TARGET_BRANCH=prod ./utils/autoMergeRequest.sh
  tags:
    - k8s-preprod-runner
  only:
    - preprod

prod:
  stage: prod
  script: 
    - echo 'deploy prod'
  tags:
    - k8s-prod-runner
  only:
    - prod

The preprod and prod branches need to be marked as protected. Go to Settings > Repository > Protected branches.

Conclusion

In this post, we created a devops cluster, we mounted the config.yml file to the specific runner, and we ended up deploying our Scaleway Elements and Kubernetes resources in an environment project.

This mechanism guarantees end-to-end security for your API Keys resources in Scaleway Elements.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I've convinced you to remove your API keys from Gitlab CI variables.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

Documentation

[1] https://www.scaleway.com/en/docs/compute/kubernetes/api-cli/creating-managing-kubernetes-lifecycle-cliv2/
[2] https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml
[3] https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
[4] https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

Top 5 ways to run your legacy containers on Google Cloud

Chabane R. — Mon, 19 Jul 2021 06:28:02 +0000

If your private environment is using Docker and you want to migrate your containers to Google Cloud, this article can help you to choose the right Google Cloud Computing service.

There are many reasons for using public cloud technology like:

increasing scalability,
improving business agility,
reducing cost,
enhancing security.

All of these motivations can turn into big challenges if you choose the wrong cloud computing service.

I have met customers trying to adapt their legacy on a Google Cloud computing service by ignoring the network, security, and devops aspects. This method is not a lift & shift migration. The lift & shift is a journey, it needs a migration plan, cloud adoption, migration path and a clear vision!

I also met customers moving their containerized application to an orchestrator thinking it was a lift & shift migration. They expected minor or no modifications or refactoring needed. This strategy is an improve and move migration and it's also a journey that requires more time.

As a last example, I met a customer wanted to move their containerized application to a serverless, networkless service while maintaining communication with their on-premises environment. A worthless strategy.

When you move to Google Cloud or any other public cloud provider, only your workloads and data will be migrated, your current network, some security patterns and devops practices will remain on your private environments. In Google Cloud, you will need to embrace new best practices and patterns if you want to see your workloads running in production with the initial reasons that motivated you to choose Google Cloud.

In this article, we take an example of a containerized application that needs to be migrated to Google Cloud taking into account many aspects such as network, security, cost and devops challenges.

Google Cloud Compute Services

I explained in a previous post Serverless is not (always) Networkless! that we can represent Google Cloud services in 2 categories:

Services deployed on the Google Cloud network (networkless)
Services deployed in your private network (infrastructure)

The first category is represented by the following computing services:

Cloud functions
Cloud Run
App Engine standard

The second category is represented by the following computing services:

App Engine flexible
Kubernetes Engine
Compute Engine

A container application can be deployed on Cloud Run, App Engine flexible, Kubernetes Engine Standard, Kubernetes Engine Autopilot or Compute Engine.

5) Compute Engine

I would say: don't migrate to Google Cloud if you plan to use Compute Engine to host your containerized application. However, if you still want to host your existing containerized application in Compute Engine, you should be aware of the following constraints:

Compute Engine is not designed to host containers, but you can use Container-Optimized OS, an operating system image that is optimized for running Docker containers.
If you want to scale your application using managed instance group (MIG), each containerized application needs to run on a separate virtual machine. It Could make the operating system overhead a significant part of your cost.
You will have to operate the entire infrastructure on your own: complex security environment, lack of flexibility, upgrade & maintenance, etc.
And other limits

4) App Engine flexible

If you want to host your existing containerized application in App Engine flexible you should be aware of the following constraints:

App Engine flexible is not a serverless service, at least one running container is required
The service requires a private network to create the instances
You can remotely log in to your running instances but they are not accessible through Compute Engine
The most expensive container service because it comes with full PaaS features like firewalls, cron jobs, canary deployment, etc.
Becomes obsolete. Google Cloud doesn't invest much in new features.

3) Kubernetes Engine Standard

If you want to host your existing containerized application in this Kubernetes Engine mode you should be aware of the following constraints:

The service requires a private network to create the nodes
You manage the cluster. So you need a solid knowledge of Kubernetes
Even if no workload is running in the cluster, you will have to pay the GKE control plane fee

2) Kubernetes Engine Autopilot

If you want to host your existing containerized application in this Kubernetes Engine mode you should be aware of the following constraints:

The service requires a private network to create the nodes
Even if no workload is running in the cluster, you will have to pay the GKE control plane fee
Suitable for hosting business applications that you control and maintain. You cannot deploy external tools like Istio (e.g Ingress Gateway)
Nodes are not accessible through Compute Engine
And other limits

1) Cloud Run

If you want to host your existing containerized application in Cloud Run you should be aware of the following constraints:

Your application should start very quickly. Because Cloud Run starts a new container for each new request (in case of cold start) [1]
The service supports only HTTP(s)
Your HTTP(s) requests cannot exceed 60 minutes
Your app does not need to communicate with a service hosted on-premises or in a Google Cloud virtual private cloud (VPC). If you use the Serverless VPC Connector with Cloud Run, you will face network and security challenges that you cannot ignore.

Recommendations

If your existing application is a Web application and does not need to communicate with a private environment, choose Cloud Run.
If your existing application does not need to communicate with a private environment (even if it's possible) and you have multiple teams and projects that need to be separated from each other, App Engine flexible is best suited for this type of organization as it is full PaaS that lets developers focus on what they do best: writing code.
If your existing application needs to communicate with a private environment and
- you need Kubernetes for basic usage, so choose Kubernetes Engine Autopilot.
- you have strong knowledge of Kubernetes, so choose Kubernetes Engine Standard.
Only if your containerized application depends on the Docker Engine, choose Compute Engine.

That's all folks!

Thanks for reading this post :-)

Documentation

[1] https://cloud.google.com/blog/topics/developers-practitioners/3-ways-optimize-cloud-run-response-times

Serverless is not (always) Networkless!

Chabane R. — Mon, 12 Jul 2021 06:47:29 +0000

"I want to put my code in a Google Cloud function and send my messages to a message broker".

A serverless application deployed in the cloud provider's network that needs to communicate with an infrastructure application deployed in a private network. It's not that easy to set up and maintain!

Serverless and infrastructure, two worlds initially designed to not work together?

Many customers on Google Cloud use Serverless services to get away from server management, network infrastructure management, and some layers of security. This is a good approach, but not always the best solution. When we start from scratch in the Cloud, choosing a service based on the Google Cloud network is indeed the best approach, but when we come from the world of private infrastructure and want to communicate the two, we will quickly realize that they are different. So when we come across a Serverless solution that requires a private network, one of the biggest challenges begins, that of making the hidden world (Networkless) communicate with the revealed world (The infrastracture).

Cloud computing services

We often group public cloud services into five parts: Infrastructure (IaaS), Container (CaaS), Platform (PaaS), Function (FaaS), and Software as a Service (SaaS).

Public cloud services that rely on these cloud computing services always try to stick to the concepts. These concepts are not always understood by customers. Some users try to mirror their legacy to these services and very often find themselves faced with security and network challenges that they did not anticipate.

To help customers guarantee this communications between these different levels of cloud computing, cloud providers have developed a specific service to establish this link. For example, Google Cloud provides a Serverless VPC Connector to allow Google Cloud Functions (FaaS), App Engine (PaaS) and Cloud Run (PaaS) to communicate privately with Kubernetes Engine containers (CaaS) and Compute Engine instances (IaaS). You will find same service for Amazon Web Services with VPC Endpoint Services or Microsoft Azure with Virtual Network service endpoints.

If you are migrating to the cloud to move away from infrastructure management and your business uses only cloud functions but needs to communicate privately with a legacy application deployed in a virtual private network (VPC), you're going to have to maintain, operate and secure a private network over the long term, and it's not that simple. The network is an important part of an infrastructure.

If you really want to move away from infrastructure management, you will also need to be Networkless.

Serverless vs Networkless

What is Serverless?

Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers. [1]

Serverless has positive impacts on the software development lifecycle while simplifying the developer experience by eliminating server management. Serverless lets you focus on business development.

Google Cloud’s serverless platform lets you build, develop, and deploy functions and applications, as source code or containers, while simplifying the developer experience by eliminating all infrastructure management. [2]

Most Google Cloud serverless services eliminate the infrastructure management like Cloud Functions, App Engine Standard or Cloud Run but not always.

Cloud Dataflow as an example is presented as a serverless service.

Dataflow: Unified stream and batch data processing that's serverless, fast, and cost-effective. [3]

For a new or not yet mature customer on Google Cloud, he will think that all the infrastructure is managed by the cloud provider, this will be the case for server mangement but not always for the network. I see many customers who use serverless for computing (Cloud functions, Cloud Run, etc.) and use Cloud Dataflow for data processing without knowing that they are building their jobs in the default network (which not recommended for production). In Google Cloud, network configuration is always presented as optional.

This is also the case for Cloud Composer, Cloud Dataproc, Cloud Kubernetes Engine Autopilot or App Engine flexible. Google Cloud will manage the servers for you, but network management will be your responsibility. Serverless services but not Networkless.

We can represent Google Cloud services in 2 categories:

Services deployed on the Google Cloud network (networkless)
Services deployed in your private network (infrastructure)

It can be illustrated like that:

Compute services

Data analytics services

Databases services

Storage services

You notice that a service may move from one location to another depending on the network configuration. Cloud SQL is a good example.

In the security overview, it makes no sense to configure a private Cloud SQL instance if you are only using Cloud functions or Cloud Run services. A public Cloud SQL instance already has two strong protections: Cloud IAM permissions and database credentials, in addition to authorized networks.

Don't be intimidated by the word private. I remind you that you are on a public cloud... ;-)

Recommendations

If you're moving to the cloud and some apps you depend on need to stay on-premise or in a Google Virtual Private Cloud (VPC), migrate all your apps to Google Compute Engine or Google Kubernetes Engine and use only Serverless services that need a private network. This design ensures an isolated network, which is a network security best practice.

The cloud migration target

So migrate your workloads in the right service with the right network.

If you are a new company (no legacy), choose only networkless services. Focus on your business and let Google Cloud manage the infrastructure for you.

Networkless services

I see a lot of new companies, where developers who build a new app in Google Cloud are reusing their existing knowledge of open source tools and frameworks to deploy it on a Google Cloud infrastructure. This strategy is not the best. If you leave the company someone else will have to maintain your infrastructure. The infrastructure is only there for existing companies and it costs!

For example, I saw a company set up a standard regional Kubernetes Engine cluster, deploy an Apache Kafka, Elasticsearch, and MongoDB containers in addition to their frontend and backend applications.

If they had deployed this to avoid the vendor lock-in (Personally against this motivation), I wouldn't have taken it as an example.

For this company's business use case, it could have used Cloud Pub / Sub, BigQuery, Firestore, Cloud Storage, and Cloud Run instead. It would have been less expensive and certainly less complex to maintain and secure!

So deploy your workloads in the right service with the right options.

That's all folks!

Thanks for reading this post :-)

Documentation

[1] https://en.wikipedia.org/wiki/Serverless_computing
[2] https://cloud.google.com/serverless
[3] https://cloud.google.com/dataflow

Securing the connectivity between a GKE application and a Cloud SQL database

Chabane R. — Thu, 24 Jun 2021 12:51:42 +0000

In the previous part we created our Cloud SQL instance. In this part, we'll put them all together and deploy Wordpress to Kubernetes and connect it to the Cloud SQL database. Our objectives are to:

Create the IAM Service Account to connect to the Cloud SQL instance. It will be associated to the Wordpress Kubernetes service account.
Create 2 deployments: One for Wordpress and one for the Cloud SQL Proxy.
Create the Cloud Armor security policy to restrict load balancer traffic to only authorized networks.
Configure the OAuth consent screen and credentials to enable Identity Aware Proxy.
Create SSL certificates and enable the HTTPs redirection.

IAM Service Account

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE.

With Workload Identity, you can configure a Kubernetes service account to act as a Google service account. Pods running as the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs.

Let's create this Google service account. Create the file infra/plan/service-account.tf.

resource "google_service_account" "web" {
  account_id   = "cloud-sql-access"
  display_name = "Service account used to access cloud sql instance"
}

resource "google_project_iam_binding" "cloudsql_client" {
  role    = "roles/cloudsql.client"
  members = [
    "serviceAccount:cloud-sql-access@${data.google_project.project.project_id}.iam.gserviceaccount.com",
  ]
}

data "google_project" "project" {
}

And the associated Kubernetes service account in infra/k8s/data/service-account.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: cloud-sql-access@<PROJECT_ID>.iam.gserviceaccount.com
  name: cloud-sql-access

Let's run our updated terraform:

cd infra/plan

terraform apply

And create the Kubernetes service account:

gcloud container clusters get-credentials private --region $REGION --project $PROJECT_ID

$ kubectl create namespace wordpress

sed -i "s/<PROJECT_ID>/$PROJECT_ID/g;" infra/k8s/data/service-account.yaml

$ kubectl create -f infra/k8s/data/service-account.yaml -n wordpress

The Kubernetes service account will be used by the Cloud SQL Proxy deployment to access the Cloud SQL instance.

Allow the Kubernetes service account to impersonate the created Google service account by an IAM policy binding between the two:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$PROJECT_ID.svc.id.goog[wordpress/cloud-sql-access]" \
  cloud-sql-access@$PROJECT_ID.iam.gserviceaccount.com

Cloud SQL Proxy

We use the Cloud SQL Auth proxy to secure access to our Cloud SQL instance without the need for Authorized networks or for configuring SSL.

Let's begin by the deployment resource:

infra/k8s/data/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cloud-sql-proxy
  name: cloud-sql-proxy
spec:
  selector:
    matchLabels:
      app: cloud-sql-proxy
  strategy: {}
  replicas: 3
  template:
    metadata:
      labels:
        app: cloud-sql-proxy
    spec:
      serviceAccountName: cloud-sql-access
      containers: 
        - name: cloud-sql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.23.0
          ports:
            - containerPort: 3306
              protocol: TCP
          envFrom:
            - configMapRef:
                name: cloud-sql-instance
          command:
            - "/cloud_sql_proxy"
            - "-ip_address_types=PRIVATE"
            - "-instances=$(CLOUD_SQL_PROJECT_ID):$(CLOUD_SQL_INSTANCE_REGION):$(CLOUD_SQL_INSTANCE_NAME)=tcp:0.0.0.0:3306"
          securityContext:
            runAsNonRoot: true
          resources:
            requests:
              memory: 2Gi
              cpu: 1

The deployment resource refers to the service account created earlier. Cloud SQL instance details are retrieved from a Kubernetes config map:

infra/k8s/data/config-map.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-sql-instance
data:
  CLOUD_SQL_INSTANCE_NAME: <CLOUD_SQL_INSTANCE_NAME>
  CLOUD_SQL_INSTANCE_REGION: <CLOUD_SQL_REGION>
  CLOUD_SQL_PROJECT_ID: <CLOUD_SQL_PROJECT_ID>

We expose the deployment resource using a Kubernetes service:

infra/k8s/data/service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cloud-sql-proxy
  name: cloud-sql-proxy
spec:
  ports:
    - port: 3306
      protocol: TCP
      name: cloud-sql-proxy
      targetPort: 3306
  selector:
    app: cloud-sql-proxy

Let's create our resources and check if the connection is established:

cd infra/plan

sed -i "s/<CLOUD_SQL_PROJECT_ID>/$PROJECT_ID/g;s/<CLOUD_SQL_INSTANCE_NAME>/$(terraform output cloud-sql-instance-name | tr -d '"')/g;s/<CLOUD_SQL_REGION>/$REGION/g;" ../k8s/data/config-map.yaml

$ kubectl create -f ../k8s/data -n wordpress

$ kubectl get pods -l app=cloud-sql-proxy -n wordpress

NAME                              READY   STATUS    RESTARTS   AGE
cloud-sql-proxy-fb9968d49-hqlwb   1/1     Running   0          4s
cloud-sql-proxy-fb9968d49-wj498   1/1     Running   0          5s
cloud-sql-proxy-fb9968d49-z95zw   1/1     Running   0          4s

$ kubectl logs cloud-sql-proxy-fb9968d49-hqlwb -n wordpress

2021/06/23 14:43:21 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2021/06/23 14:43:25 Listening on 0.0.0.0:3306 for <PROJECT_ID>:<REGION>:<CLOUD_SQL_INSTANCE_NAME>
2021/06/23 14:43:25 Ready for new connections

Ok! Let's move on to the Wordpress application

Wordpress application

Let's begin by the deployment resource:

infra/k8s/web/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
        - image: wordpress
          name: wordpress
          env:
          - name: WORDPRESS_DB_HOST
            value: cloud-sql-proxy:3306
          - name: WORDPRESS_DB_USER
            value: wordpress
          - name: WORDPRESS_DB_NAME
            value: wordpress
          - name: WORDPRESS_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql
                key: password
          ports:
            - containerPort: 80
              name: wordpress
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
          livenessProbe:
            initialDelaySeconds: 30
            httpGet:
              port: 80
              path: /wp-admin/install.php # at the very beginning, this is the only accessible page. Don't forget to change to /wp-login.php 
          readinessProbe:
            httpGet:
              port: 80
              path: /wp-admin/install.php
          resources:
            requests:
              cpu: 1000m
              memory: 2Gi
            limits:
              cpu: 1200m
              memory: 2Gi
      volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wordpress

infra/k8s/web/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: wordpress
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: wordpress

The cloud.google.com/neg annotation specifies that port 80 will be associated with a zonal network endpoint group (NEG). See Container-native load balancing for information on the benefits, requirements, and limitations of container-native load balancing.

We create a PVC for Wordpress:

infra/k8s/web/volume-claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

We finish this section by initializing our Kubernetes ingress resource. This resource will allow us to access the Wordpress application from the internet.

Create the file infra/k8s/web/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "wordpress"
    kubernetes.io/ingress.class: "gce"
  name: wordpress
spec:
  defaultBackend:
    service:
      name: wordpress
      port:
        number: 80
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: wordpress
            port:
              number: 80

The kubernetes.io/ingress.global-static-ip-name annotation specifies the name of the global IP address resource to be associated with the HTTP(S) Load Balancer. [2]

Let's create our resources and test the Wordpress application:

cd infra/k8s

$ kubectl create secret generic mysql \
    --from-literal=password=$(gcloud secrets versions access latest --secret=wordpress-admin-user-password --project $PROJECT_ID) -n wordpress

gcloud compute addresses create wordpress --global

$ kubectl create -f web -n wordpress

$ kubectl get pods -l app=wordpress -n wordpress

NAME                         READY   STATUS    RESTARTS   AGE
wordpress-6d58d85845-2d7x2   1/1     Running   0          10m

$ kubectl get ingress -n wordpress
NAME        CLASS    HOSTS   ADDRESS         PORTS   AGE
wordpress   <none>   *       34.117.187.51   80      16m

Anyone can have access to the application. Let's create a Cloud Armor security policy to restrict traffic to the only authorized network.

Cloud Armor security policy

We use Cloud Armor security policy to filter incoming traffic that is destined to external HTTP(S) load balancers.

Create the ìnfra/plan/cloud-armor.tf:

resource "google_compute_security_policy" "wordpress" {
  name = "wordpress"

  rule {
    action   = "allow"
    priority = "1000"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = var.authorized_source_ranges
      }
    }
    description = "Allow access to authorized source ranges"
  }

  rule {
    action   = "deny(403)"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
    description = "default rule"
  }

}

Let's run our updated terraform:

cd infra/plan

terraform apply

Now, let's create a backend config in Kubernetes and reference the security policy.

BackendConfig custom resource definition (CRD) allows us to further customize the load balancer. This CRD allows us to define additional load balancer features hierarchically, in a more structured way than annotations. [3]

infra/k8s/web/backend.yaml

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: wordpress
spec:
  securityPolicy:
    name: wordpress

kubectl create -f infra/k8s/web/backend.yaml -n wordpress

Add the annotation cloud.google.com/backend-config: '{"default": "wordpress"}' in the wordpress service:

kubectl apply -f infra/k8s/web/service.yaml -n wordpress

Let's check if the HTTP Load balancer has attached the security policy:

It looks nice. Let's do a test.

Put a bad IP to test if we are rejected

gcloud compute security-policies rules update 1000 \
    --security-policy wordpress \
    --src-ip-ranges "85.56.40.96"

curl http://34.117.187.51/

<!doctype html><meta charset="utf-8"><meta name=viewport content="width=device-width, initial-scale=1"><title>403</title>403 Forbidden

Let's put a correct IP

gcloud compute security-policies rules update 1000 \
    --security-policy wordpress \
    --src-ip-ranges $(curl -s http://checkip.amazonaws.com/)

curl http://34.117.187.51/
<!doctype html>
<html lang="en-GB" >
<head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <title>admin &#8211; Just another WordPress site</title>

Ok!

With this Backend configuration, only employees in the same office can have access to the application. Now we want them to be authenticated to access the application as well. We can achieve this by using Identity Aware Proxy (Cloud IAP).

Enabling Cloud IAP

We use IAP to establish a central authorization layer for our Wordpress application accessed by HTTPS.

IAP is integrated through Ingress for GKE. This integration enables you to control resource-level access for employees instead of using a VPN. [1]

Follow the instructions described in the GCP documentation to:

Create a Kubernetes secret to wrap the OAuth client you created earlier:

CLIENT_ID_KEY=<CLIENT_ID_KEY>
CLIENT_SECRET_KEY=<CLIENT_SECRET_KEY>
kubectl create secret generic wordpress --from-literal=client_id=$CLIENT_ID_KEY \
    --from-literal=client_secret=$CLIENT_SECRET_KEY \
    -n wordpress

Let's update our Backend configuration

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: wordpress
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: wordpress
  securityPolicy:
    name: wordpress

Apply the changes:

kubectl apply -f infra/k8s/web/backend-config.yaml -n wordpress

Let's do a test

Ok!

SSL Certificates

If you have a domain name, you can enable Google-managed SSL certificates using the CRD ManagedCertificate.

Google-managed SSL certificates are Domain Validation (DV) certificates that Google Cloud obtains and manages for your domains. They support multiple hostnames in each certificate, and Google renews the certificates automatically. [4]

Create the file infra/k8s/web/ssl.yaml

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: wordpress
spec:
  domains:
    - <DOMAIN_NAME>

We can create the domain name using Terraform or simply with the gcloud command:

export PUBLIC_DNS_NAME=
export PUBLIC_DNS_ZONE_NAME=

gcloud dns record-sets transaction start --zone=$PUBLIC_DNS_ZONE_NAME
gcloud dns record-sets transaction add $(gcloud compute addresses list --filter=name=wordpress --format="value(ADDRESS)") --name=wordpress.$PUBLIC_DNS_NAME. --ttl=300 --type=A --zone=$PUBLIC_DNS_ZONE_NAME
gcloud dns record-sets transaction execute --zone=$PUBLIC_DNS_ZONE_NAME

sed -i "s/<DOMAIN_NAME>/wordpress.$PUBLIC_DNS_NAME/g;" infra/k8s/web/ssl.yaml

kubectl create -f infra/k8s/web/ssl.yaml -n wordpress

Add the annotation networking.gke.io/managed-certificates: "wordpress" in your ingress resource.

Let's do a test

Ok!

To redirect all HTTP traffic to HTTPS, we need to create a FrontendConfig.

Create the file infra/k8s/web/frontend-config.yaml

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: wordpress
spec:
  redirectToHttps:
    enabled: true
    responseCodeName: MOVED_PERMANENTLY_DEFAULT

kubectl create -f infra/k8s/web/frontend-config.yaml -n wordpress

Add the annotation networking.gke.io/v1beta1.FrontendConfig: "wordpress" in your ingress resource.

Let's do a test

curl -s http://wordpress.<HIDDEN>.stack-labs.com/

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://wordpress.<HIDDEN>.stack-labs.com/">here</A>.
</BODY></HTML>

Ok then!

Conclusion

Congratulations! You have completed this long workshop. In this series we have:

Created an isolated network to host our Cloud SQL instance
Configured an Google Kubernetes Engine Autopilot cluster with fine-grained access control to Cloud SQL instance
Tested the connectivity between a Kubernetes container and a Cloud SQL instance database.
Secured the access to the Wordpress application

That's it!

Clean

Remove the NEG resources. You will find them in Compute Engine > Network Endpoint Group.

Run the following commands:

terraform destroy 

gcloud dns record-sets transaction remove $(gcloud compute addresses list --filter=name=wordpress --format="value(ADDRESS)") --name=wordpress.$PUBLIC_DNS_NAME. --ttl=300 --type=A --zone=$PUBLIC_DNS_ZONE_NAME
gcloud dns record-sets transaction execute --zone=$PUBLIC_DNS_ZONE_NAME

gcloud compute addresses delete wordpress

gcloud secrets delete wordpress-admin-user-password

Final Words

The source code is available on Gitlab.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I have helped you answer some of the hard questions about connecting GKE Autopilot to Cloud SQL and providing a pod level defense in depth security strategy at both the networking and authentication layers.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

Documentation

[1] https://cloud.google.com/iap/docs/enabling-kubernetes-howto
[2] https://cloud.google.com/kubernetes-engine/docs/tutorials/configuring-domain-name-static-ip
[3] https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#configuring_ingress_features
[4] https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs

Securing sensitive data in Cloud SQL

Chabane R. — Wed, 23 Jun 2021 12:07:29 +0000

In the previous part we created our GKE Autopilot cluster. In this part we will configure the Cloud SQL Instance.

The following resources will be created:

A highly available private Cloud SQL MySQL Instance
A database and a user
An automatic internal IP range for private connection
A private connection to GCP services

Cloud SQL

The Cloud SQL Instance used is a MySQL database server
The Multiples zones option is enabled to ensure high-availability
The Instance is not publicly accessible and it's reachable only using its private IP
The authentication is done via IAM
Automated backup is enabled
We create a database and a user for later

Create a terraform file infra/plan/cloud-sql.tf

resource "random_string" "db_name_suffix" {
  length  = 4
  special = false
  upper   = false
}

resource "google_sql_database_instance" "mysql" {

  # Instance info
  name             = "mysql-private-${random_string.db_name_suffix.result}"
  region           = var.region
  database_version = var.mysql_database_version

  settings {

    # Region and zonal availability
    availability_type = var.mysql_availability_type
    location_preference {
      zone = var.mysql_location_preference
    }

    # Machine Type
    tier              = var.mysql_machine_type

    # Storage
    disk_size         = var.mysql_default_disk_size

    # Connections
    ip_configuration {
      ipv4_enabled        = false
      private_network     = google_compute_network.custom.id
    }

    # Backups
    backup_configuration {
      binary_log_enabled = true
      enabled = true
      start_time = "06:00"
    }
  }
  depends_on = [
    google_service_networking_connection.private-vpc-connection
  ]
}

data "google_secret_manager_secret_version" "wordpress-admin-user-password" {
  secret = "wordpress-admin-user-password"
}

resource "google_sql_database" "wordpress" {
  name     = "wordpress"
  instance = google_sql_database_instance.mysql.name
}

resource "google_sql_user" "wordpress" {
  name = "wordpress"
  instance = google_sql_database_instance.mysql.name
  password = data.google_secret_manager_secret_version.wordpress-admin-user-password.secret_data
}

Add the following outputs

output "cloud-sql-connection-name" {
  value = google_sql_database_instance.mysql.connection_name
}

output "cloud-sql-instance-name" {
  value = "mysql-private-${random_string.db_name_suffix.result}"
}

Private connection

We need to configure private services access to allocate an IP address range and create a private service connection. This will allow resources in the Web subnet to connect to the Cloud SQL instance.

Complete the file infra/plan/vpc.tf with the following resources:

resource "google_compute_global_address" "private-ip-peering" {
  name          = "google-managed-services-custom"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 24
  network       = google_compute_network.custom.id
}

resource "google_service_networking_connection" "private-vpc-connection" {
  network = google_compute_network.custom.id
  service = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [
    google_compute_global_address.private-ip-peering.name
  ]
}

Complete the file infra/plan/variable.tf:

variable "region" {
  type = string
  default = "europe-west1"
}

variable "mysql_location_preference" {
  type = string
  default = "europe-west1-b"
}

variable "mysql_machine_type" {
  type = string
  default = "db-n1-standard-2"
}

variable "mysql_database_version" {
  type = string
  default = "MYSQL_8_0"
}

variable "mysql_default_disk_size" {
  type = string
  default = "100"
}

variable "mysql_availability_type" {
  type = string
  default = "REGIONAL"
}

Before applying the changes, we need to create the secret of the user password:

gcloud services enable secretmanager.googleapis.com --project $PROJECT_ID

gcloud beta secrets create wordpress-admin-user-password --locations $REGION --replication-policy user-managed
echo -n "changeme" | gcloud beta secrets versions add wordpress-admin-user-password --data-file=-

Let's deploy our Cloud SQL instance

cd infra/plan

gcloud services enable sqladmin.googleapis.com --project $PROJECT_ID

terraform apply

Let's check if all the resources have been created and are working correctly:

Cloud SQL instance

Peering connection

Private connection

Conclusion

Our Cloud SQL instance is now available. In the last part, we'll establish a connection between a container deployed in GKE cluster and a database created in an Cloud SQL instance.

Creating a Google Kubernetes Engine Autopilot cluster using Terraform

Chabane R. — Wed, 23 Jun 2021 09:29:01 +0000

In the previous part we created our network stack. In this part we will configure the GKE Autopilot cluster.

The following resources will be created:

GKE Autopilot Cluster

GKE Autopilot Cluster

Our GKE Autopilot Cluster is hosted in the Web subnet. The public API server endpoint can only be accessed from a specific range of IP addresses.

Create the terraform file infra/plan/gke.tf:

resource "google_container_cluster" "private" {
  provider                 = google-beta

  name                     = "private"
  location                 = var.region

  network                  = google_compute_network.custom.name
  subnetwork               = google_compute_subnetwork.web.id

  private_cluster_config {
    enable_private_endpoint = false
    enable_private_nodes    = true
    master_ipv4_cidr_block  = var.gke_master_ipv4_cidr_block
  }

  master_authorized_networks_config {
    dynamic "cidr_blocks" {
        for_each = var.authorized_source_ranges
        content {
            cidr_block = cidr_blocks.value
        }
    }
   }

  maintenance_policy {
    recurring_window {
      start_time = "2021-06-18T00:00:00Z"
      end_time   = "2050-01-01T04:00:00Z"
      recurrence = "FREQ=WEEKLY"
    }
  }

  # Enable Autopilot for this cluster
  enable_autopilot = true

  # Configuration of cluster IP allocation for VPC-native clusters
  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  # Configuration options for the Release channel feature, which provide more control over automatic upgrades of your GKE clusters.
  release_channel {
    channel = "REGULAR"
  }
}

Complete the file infra/plan/variable.tf:

variable "gke_master_ipv4_cidr_block" {
  type    = string
  default = "172.23.0.0/28"
}

Let's deploy our cluster

cd infra/plan

gcloud services enable container.googleapis.com --project $PROJECT_ID
terraform apply

Let's check if the cluster has been created and is working correctly:

GKE Autopilot cluster

Conclusion

Our GKE cluster is now active. In the next part, we will focus on setting up the Cloud SQL instance.

Configuring an isolated network in Google Cloud

Chabane R. — Wed, 23 Jun 2021 08:54:36 +0000

In the first part we introduced the security patterns that could be implemented to secure the connectivity between Google Kubernetes Engine and Cloud SQL. In this part we will implement the network isolation by deploying the following GCP resources:

VPC with 2 subnets
- 1 web subnet for GKE.
- 1 data subnet.
Cloud NAT attached to the web subnet.
Firewall rules to restrict access to subnets to only authorized networks.

It is recommended to group similar applications into fewer, more manageable and larger subnets.

If you have multiples GKE clusters per environment, Google Cloud recommends to use Shared VPC to reduce management and topology complexity.

VPC

Let's start with the Virtual Private Cloud.

Create a terraform file infra/plan/vpc.tf:

A simple VPC resource
The web subnet. It will host our Google Kubernetes Engine
The data subnet. It could host your Cloud Dataflow jobs, Cloud Composer environments, etc.

resource "google_compute_network" "custom" {
  name                    = "custom"
  auto_create_subnetworks = "false" 
  routing_mode            = "GLOBAL"
}

resource "google_compute_subnetwork" "web" {
  name          = "web"
  ip_cidr_range = "10.10.10.0/24"
  network       = google_compute_network.custom.id
  region        = var.region

  secondary_ip_range  = [
    {
        range_name    = "services"
        ip_cidr_range = "10.10.11.0/24"
    },
    {
        range_name    = "pods"
        ip_cidr_range = "10.1.0.0/20"
    }
  ]

  private_ip_google_access = true
}

resource "google_compute_subnetwork" "data" {
  name          = "data"
  ip_cidr_range = "10.20.10.0/24"
  network       = google_compute_network.custom.id
  region        = var.region

  private_ip_google_access = true
}

Cloud NAT

In order to allow our web subnet used by GKE to access the internet, we need to create a Cloud NAT. We associate Cloud NAT with the subnet using Cloud Router.

Create a terraform file infra/plan/nat.tf

resource "google_compute_address" "web" {
  name    = "web"
  region  = var.region
}

resource "google_compute_router" "web" {
  name    = "web"
  network = google_compute_network.custom.id
}

resource "google_compute_router_nat" "web" {
  name                               = "web"
  router                             = google_compute_router.web.name
  nat_ip_allocate_option             = "MANUAL_ONLY"
  nat_ips                            = [ google_compute_address.web.self_link ]
  source_subnetwork_ip_ranges_to_nat = "LIST_OF_SUBNETWORKS" 
  subnetwork {
    name                    = google_compute_subnetwork.web.id
    source_ip_ranges_to_nat = ["ALL_IP_RANGES"]
  }
  depends_on                         = [ google_compute_address.web ]
}

Firewall

Firewall allows us to restrict the inbound and outbound network traffic to and from a VM instance, a network tag or a service account. In our case, we can implement the following rules:

A rule to restrict access to the Cloud SQL MySQL instance port to only GKE nodes.
A rule to restrict network access to only authorized networks.

Create a terraform file infra/plan/firewall.tf

resource "google_compute_firewall" "mysql" {
  name    = "allow-only-gke-cluster"
  network = google_compute_network.custom.name

  allow {
    protocol = "tcp"
    ports    = ["3306"]
  }

  priority = 1000

  source_ranges = ["10.10.10.0/24"]
}

resource "google_compute_firewall" "web" {
  name    = "allow-only-authorized-networks"
  network = google_compute_network.custom.name

  allow {
    protocol = "tcp"
  }

  priority = 1000

  source_ranges = var.authorized_source_ranges
}

Let's configure our terraform.

Create a terraform file infra/plan/variable.tf

variable "region" {
  type = string
  default = "europe-west1"
}

variable "authorized_source_ranges" {
  type        = list(string)
  description = "Addresses or CIDR blocks which are allowed to connect to GKE API Server."
}

Add a infra/plan/version.tf file

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "3.71.0"
    }
  }
}

Add a infra/plan/provider.tf file

provider "google" {
  region  = "europe-west1"
}

And a infra/plan/backend.tf

terraform {
  backend "gcs" {
  }
}

Now, export the following variables and create a bucket to save your terraform states.

export PROJECT_ID=<PROJECT_ID>
export REGION=<REGION>
export TERRAFORM_BUCKET_NAME=<BUCKET_NAME>

gcloud config set project ${PROJECT_ID}

gsutil mb -c standard -l ${REGION} gs://${TERRAFORM_BUCKET_NAME}
gsutil versioning set on gs://${TERRAFORM_BUCKET_NAME}

Create a infra/plan/terraform.tfvars and deploy the infrastructure:

authorized_source_ranges    = ["<AUTHORIZED_NETWORK>"]

cd infra/plan

sed -i "s,<AUTHORIZED_NETWORK>,$AUTHORIZED_NETWORK,g" terraform.tfvars

terraform init \
  -backend-config="bucket=${TERRAFORM_BUCKET_NAME}" \
  -backend-config="prefix=state"

terraform apply

Let's check if all the resources have been created and are working correctly

VPC & Subnets

Cloud NAT

Firewalls

Conclusion

Our network is now ready to host our GCP resources. In the next part, we will focus on setting up GKE Autopilot.

Designing a defense-in-depth network security model between Google Kubernetes Engine and Cloud SQL

Chabane R. — Tue, 15 Jun 2021 14:57:28 +0000

It is very easy today to establish a connection between a container in Kubernetes and a relational database server, just create a SQL user and open a TCP connection. In cloud computing, in the case of Google Cloud Platform, the equivalent is connecting a container in a Google Kubernetes Engineer cluster to a Cloud SQL instance.

Important points should be taken into account in setting up this connectivity.

Which network topology to choose? How to authenticate and authorize the connection to the Cloud SQL instance? Can I publicly expose a private Cloud SQL instance?

Which architecture could be the most efficient, maintainable and scalable?

Scenarios

Cloud SQL supports the following scenarios for accessing a DB instance in a VPC:

A Compute Engine instance in the same VPC
A Compute Engine in a different VPC
A client application through the internet
A private network

The scenarios that concern us are the first two:

GKE and Cloud SQL in the same VPC.
GKE and Cloud SQL in different VPCs.

In the first one, there is a direct communication between Kubernetes workloads and Cloud SQL instances.

In the second scenario, if private communication is required, a VPN connection between the two VPCs must be established.

Let's discover the possible architectures that could be used to implement each scenario.

Direct communication with public IP and authorized networks

In this architecture [2], our Cloud SQL instance is isolated on its own subnet and accessible through public IP address to only GKE Autopilot that requires access to it. Pods have access to Cloud SQL using Cloud SQL Proxy.

Direct communication with private IP

In this architecture [3], our Cloud SQL instance is isolated on its own subnet and accessible with a private IP to only GKE Autopilot that requires access to it. Pods have access to Cloud SQL using Cloud SQL Proxy.

Communication using VPN

In this architecture, our Cloud SQL instance is isolated on its own VPC. The two VPCs communicate using Cloud VPN. Pods have access to Cloud SQL using Cloud SQL Proxy.

Each architecture has its own advantages and disadvantages but all apply network isolation best practices for securing sensitive data in Cloud SQL.

Let's explore scenario 2.

Scenario 2 in detail

In the scenario 2 architecture, the network isolation is achieved using Private services access. We can go more deeply using GKE Workload Identity add-on to provide a pod level defense in depth security strategy at both the networking and authentication layers.

We associate a Google Service Account with a Kubernetes service account. This service account can then provide IAM permissions to the containers in any pod that uses that service account.
Network policies to define rules that allow inbound and outbound network traffic to and from pods.

We could implement the same security pattern with the scenario of Cloud VPN.

Now that we have a clear idea of the concepts, let's implement this architecture.

Prerequisites

Download, install, and configure the Google Cloud SDK.
Terraform
Kubectl

Architecture

The overall architecture that we will implement during this series of articles is as follows:

Objectives

During this section of the workshop:

With Terraform

We will create a VPC with 2 subnets
- web subnet for GKE Autopilot.
- data subnet for the Cloud SQL instance.
NAT gateway attached to web subnet, but not on data subnet as Cloud SQL doesn't need to access the public internet.
GKE Autopilot.
A highly available Cloud SQL MySQL Instance.

With Kubectl

We will create an annotated Kubernetes service account with Google Service Account that has the necessary permission to connect to the Cloud SQL instance.
Cloud SQL proxy.
A Web application. A Kubernetes deployment to connect with our MySQL database with:
- A GKE ingress, to create an external HTTPS Load balancer
- A managed SSL certificate.
- A backend config with IAP and Cloud Armor security policy enabled.
- A frontend config to redirect http requests to https.

The series is divided into five parts:

Configuring an isolated network in Google Cloud
Creating a GKE Autopilot cluster using Terraform
Securing sensitive data in Cloud SQL
Securing the connectivity between a GKE application and a Cloud SQL database

Conclusion

In this first part, we discussed possible scenarios for securing communication between GKE workloads and Cloud SQL databases. In the next section, we'll implement our network stack using Terraform.

Documentation

[1] https://medium.com/google-cloud/cloud-sql-with-private-ip-only-the-good-the-bad-and-the-ugly-de4ac23ce98a
[2] https://cloud.google.com/sql/docs/mysql/configure-ip
[3] https://cloud.google.com/sql/docs/mysql/private-ip

Deploying production-ready GitLab on Amazon EKS with Terraform

Chabane R. — Mon, 03 May 2021 08:50:40 +0000

Gitlab is an open DevOps platform. You can use the SaaS version where there is no technical setup required or you can download, install and maintain your own GitLab self-managed, on your own infrastructure or in the public cloud environment. See the key differences between GitLab SaaS & self-managed

In this blog post we will choose the GitLab self-managed version and deploy the instance using the official Helm chart. This chart is the recommended, and supported method to install GitLab on a cloud native environment.

The gitlab/gitlab chart is the best way to operate GitLab on Kubernetes. This chart contains all the required components to get started, and can scale to large deployments.

By default, the helm chart deploys all components on Kubernetes. In a cloud native environment, to be production ready, we need to use cloud provided solution to outsource the dependencies.

PostgreSQL on Amazon RDS,
Redis on Amazon Elasticache,
Object storages with Amazon S3,
Use an external volume for Gitaly.

Prerequisites

Installing and configuring AWS CLI
Terraform
Kubectl
Helm cli
Create a public hosted zone in Route 53. See tutorial
Request a public certificate with AWS Certificate Manager. See tutorial

Network

In this section, we create a VPC, 2 private and public subnets, 2 NAT Gateways and an internet gateway.

plan/vpc.tf

resource "aws_vpc" "devops" {
  cidr_block           = var.vpc_cidr_block
  instance_tenancy     = "default"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Environment = "core"
    Name        = "devops"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "aws_default_security_group" "defaul" {
    vpc_id = aws_vpc.devops.id
}

plan/subnet.tf

resource "aws_subnet" "private" {
  for_each = {
    for subnet in local.private_nested_config : subnet.name => subnet
  }

  vpc_id                  = aws_vpc.devops.id
  cidr_block              = each.value.cidr_block
  availability_zone       = var.az[index(local.private_nested_config, each.value)]
  map_public_ip_on_launch = false

  tags = {
    Environment                       = "devops"
    Name                              = each.value.name
    "kubernetes.io/role/internal-elb" = 1
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "aws_subnet" "public" {
  for_each = {
    for subnet in local.public_nested_config : subnet.name => subnet
  }

  vpc_id                  = aws_vpc.devops.id
  cidr_block              = each.value.cidr_block
  availability_zone       = var.az[index(local.public_nested_config, each.value)]
  map_public_ip_on_launch = true

  tags = {
    Environment              = "devops"
    Name                     = each.value.name
    "kubernetes.io/role/elb" = 1
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

plan/nat.tf

resource "aws_eip" "nat" {
  for_each = {
    for subnet in local.public_nested_config : subnet.name => subnet
  }

  vpc = true

  tags = {
    Environment = "core"
    Name        = "eip-${each.value.name}"
  }
}

resource "aws_nat_gateway" "nat-gw" {
  for_each = {
    for subnet in local.public_nested_config : subnet.name => subnet
  }

  allocation_id = aws_eip.nat[each.value.name].id
  subnet_id     = aws_subnet.public[each.value.name].id

  tags = {
    Environment = "core"
    Name        = "nat-${each.value.name}"
  }
}

resource "aws_route_table" "private" {
  for_each = {
    for subnet in local.public_nested_config : subnet.name => subnet
  }

  vpc_id = aws_vpc.devops.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat-gw[each.value.name].id
  }

  tags = {
    Environment = "core"
    Name        = "rt-${each.value.name}"
  }
}

resource "aws_route_table_association" "private" {

  for_each = {
    for subnet in local.private_nested_config : subnet.name => subnet
  }

  subnet_id      = aws_subnet.private[each.value.name].id
  route_table_id = aws_route_table.private[each.value.associated_public_subnet].id
}

plan/igw.tf

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.devops.id

  tags = {
    Environment = "core"
    Name        = "igw-security"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.devops.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Environment = "core"
    Name        = "rt-public-security"
  }
}

resource "aws_route_table_association" "public" {
  for_each = {
    for subnet in local.public_nested_config : subnet.name => subnet
  }

  subnet_id      = aws_subnet.public[each.value.name].id
  route_table_id = aws_route_table.public.id
}

Amazon EKS

In this section we create our Kubernetes cluster with the following settings:

restrict access to a specific IP (it could be your office range IPs) and to the NAT gateways IPs (to use gitlab runners)
enable all logs
enable IAM roles for service accounts
security groups for the cluster

plan/eks-cluster.tf

resource "aws_eks_cluster" "devops" {
  name     = var.eks_cluster_name
  role_arn = aws_iam_role.eks.arn

  version  = "1.18"

  vpc_config {
    security_group_ids      = [aws_security_group.eks_cluster.id]
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = concat([var.authorized_source_ranges], [for n in aws_eip.nat : "${n.public_ip}/32"])
    subnet_ids              = concat([for s in aws_subnet.private : s.id], [for s in aws_subnet.public : s.id])
  }

  enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.eks-AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.eks-AmazonEKSVPCResourceController,
    aws_iam_role_policy_attachment.eks-AmazonEKSServicePolicy
  ]

  tags = {
    Environment = "core"
  }
}

resource "aws_iam_role" "eks" {
  name = var.eks_cluster_name

  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "Service": "eks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
        }
    ]
}
  EOF
}

data "tls_certificate" "cert" {
  url = aws_eks_cluster.devops.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "openid" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.cert.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.devops.identity[0].oidc[0].issuer
}

resource "aws_iam_role_policy_attachment" "eks-AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks.name
}

resource "aws_iam_role_policy_attachment" "eks-AmazonEKSServicePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks.name
}

# Enable Security Groups for Pods
resource "aws_iam_role_policy_attachment" "eks-AmazonEKSVPCResourceController" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks.name
}

resource "aws_security_group" "eks_cluster" {
  name        = "${var.eks_cluster_name}/ControlPlaneSecurityGroup"
  description = "Communication between the control plane and worker nodegroups"
  vpc_id      = aws_vpc.devops.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.eks_cluster_name}/ControlPlaneSecurityGroup"
  }
}

resource "aws_security_group_rule" "cluster_inbound" {
  description              = "Allow unmanaged nodes to communicate with control plane (all ports)"
  from_port                = 0
  protocol                 = "-1"
  security_group_id        = aws_eks_cluster.devops.vpc_config[0].cluster_security_group_id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 0
  type                     = "ingress"
}

Here we create two nodegroups, one private and one public.

plan/eks-nodegroup.tf

resource "aws_eks_node_group" "private" {
  cluster_name    = aws_eks_cluster.devops.name
  node_group_name = "private"
  node_role_arn   = aws_iam_role.node-group.arn
  subnet_ids      = [for s in aws_subnet.private : s.id]

  labels          = {
    "type" = "private"
  }

  instance_types = ["m5.xlarge"]

  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 2
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
  # Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
  depends_on = [
    aws_iam_role_policy_attachment.node-group-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node-group-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node-group-AmazonEC2ContainerRegistryReadOnly
  ]

  tags = {
    Environment = "core"
  }
}

resource "aws_eks_node_group" "public" {
  cluster_name    = aws_eks_cluster.devops.name
  node_group_name = "public"
  node_role_arn   = aws_iam_role.node-group.arn
  subnet_ids      = [for s in aws_subnet.public : s.id]

  labels          = {
    "type" = "public"
  }

  instance_types = ["t3.medium"]

  scaling_config {
    desired_size = 1
    max_size     = 3
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.node-group-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node-group-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node-group-AmazonEC2ContainerRegistryReadOnly,
  ]

  tags = {
    Environment = "core"
  }
}

resource "aws_iam_role" "node-group" {
  name = "eks-node-group-role-devops"

  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "node-group-AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.node-group.name
}

resource "aws_iam_role_policy_attachment" "node-group-AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.node-group.name
}

resource "aws_iam_role_policy_attachment" "node-group-AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.node-group.name
}

resource "aws_iam_role_policy" "node-group-ClusterAutoscalerPolicy" {
  name = "eks-cluster-auto-scaler"
  role = aws_iam_role.node-group.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:DescribeLaunchConfigurations",
            "autoscaling:DescribeTags",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:TerminateInstanceInAutoScalingGroup"
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role_policy" "node-group-AmazonEKS_EBS_CSI_DriverPolicy" {
  name = "AmazonEKS_EBS_CSI_Driver_Policy"
  role = aws_iam_role.node-group.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
            "ec2:AttachVolume",
            "ec2:CreateSnapshot",
            "ec2:CreateTags",
            "ec2:CreateVolume",
            "ec2:DeleteSnapshot",
            "ec2:DeleteTags",
            "ec2:DeleteVolume",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeInstances",
            "ec2:DescribeSnapshots",
            "ec2:DescribeTags",
            "ec2:DescribeVolumes",
            "ec2:DescribeVolumesModifications",
            "ec2:DetachVolume",
            "ec2:ModifyVolume"
        ]
        Effect   = "Allow"
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role" "ebs-csi-controller" {
  name = "AmazonEKS_EBS_CSI_DriverRole"

  assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": aws_iam_openid_connect_provider.openid.arn
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "${replace(aws_iam_openid_connect_provider.openid.url, "https://", "")}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                    }
                }
            }
        ]
    })
}

resource "aws_security_group" "eks_nodes" {
  name        = "${var.eks_cluster_name}/ClusterSharedNodeSecurityGroup"
  description = "Communication between all nodes in the cluster"
  vpc_id      = aws_vpc.devops.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    self        = true
  }

  ingress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    security_groups = [aws_eks_cluster.devops.vpc_config[0].cluster_security_group_id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.eks_cluster_name}/ClusterSharedNodeSecurityGroup"
    Environment = "core"
  }
}

Gitlab

Let's start by outsourcing object storages.

External object storage

GitLab relies on object storage for highly-available persistent data in Kubernetes. For production quality deployments, Gitlab recommends using a hosted object storage like Amazon S3. [1]

In the following terraform we outsource:

Docker registry images,
Long file storage, artifacts, uploads, packages, external pseudonymizer,
Backups.

plan/s3.tf


resource "aws_route53_record" "gitlab" {
 zone_id    = data.aws_route53_zone.public.zone_id
 name       = "gitlab.${var.public_dns_name}"
 type       = "CNAME"
 ttl        = "300"
 records    = [data.kubernetes_service.gitlab-webservice.status.0.load_balancer.0.ingress.0.hostname]

 depends_on = [
   helm_release.gitlab,
   data.kubernetes_service.gitlab-webservice
 ]
}

resource "aws_s3_bucket" "gitlab-registry" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-registry"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Registry"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-runner-cache" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-runner-cache"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Runner Cache"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-backups" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-backups"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Backups"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-pseudo" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-pseudo"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Pseudo"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "git-lfs" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-git-lfs"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Git Large File Storage"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-artifacts" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-artifacts"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Artifacts"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-uploads" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-uploads"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Uploads"
    Environment = "core"
  }
}

resource "aws_s3_bucket" "gitlab-packages" {
  bucket = "${data.aws_caller_identity.current.account_id}-${var.region}-gitlab-packages"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }

  tags = {
    Name        = "Gitlab Packages"
    Environment = "core"
  }
}

We used Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3). You can easily replace this encryption by Server-Side Encryption with Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS) [2]

IAM permissions

To authorize Gitlab to access the object storages, we need to create an IAM role.

plan/iam.tf

resource "aws_iam_role" "gitlab-access" {
  name = "gitlab-access"

  assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": aws_iam_openid_connect_provider.openid.arn
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "${replace(aws_iam_openid_connect_provider.openid.url, "https://", "")}:sub": "system:serviceaccount:gitlab:aws-access"
                    }
                }
            }
        ]
    })
}

resource "aws_iam_role_policy" "gitlab-access" {
  name = "gitlab-access"
  role = aws_iam_role.gitlab-access.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads"
        ]
        Effect   = "Allow"
        Resource = [
            aws_s3_bucket.gitlab-backups.arn,
            aws_s3_bucket.gitlab-registry.arn,
            aws_s3_bucket.gitlab-runner-cache.arn,
            aws_s3_bucket.gitlab-pseudo.arn,
            aws_s3_bucket.git-lfs.arn,
            aws_s3_bucket.gitlab-artifacts.arn,
            aws_s3_bucket.gitlab-uploads.arn,
            aws_s3_bucket.gitlab-packages.arn
        ]
      },
      {
        Action = [
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload"
        ]
        Effect   = "Allow"
        Resource = [
            "${aws_s3_bucket.gitlab-backups.arn}/*",
            "${aws_s3_bucket.gitlab-registry.arn}/*",
            "${aws_s3_bucket.gitlab-runner-cache.arn}/*",
            "${aws_s3_bucket.gitlab-pseudo.arn}/*",
            "${aws_s3_bucket.git-lfs.arn}/*",
            "${aws_s3_bucket.gitlab-artifacts.arn}/*",
            "${aws_s3_bucket.gitlab-uploads.arn}/*",
            "${aws_s3_bucket.gitlab-packages.arn}/*"
        ]
      }
    ]
  })
}

External PostgreSQL database

PostgreSQL persists the GitLab database data. We create the RDS instance with two read replicas to load balance the traffic. We restrict incoming network traffic only from private subnets.

plan/rds.tf

resource "random_string" "db_suffix" {
  length = 4
  special = false
  upper = false
}

resource "random_password" "db_password" {
  length = 12
  special = true
  upper = true
}

resource "aws_db_instance" "gitlab-primary" {

  # Engine options
  engine                              = "postgres"
  engine_version                      = "12.5"

  # Settings
  name                                = "gitlabhq_production"
  identifier                          = "gitlab-primary"

  # Credentials Settings
  username                            = "gitlab"
  password                            = "p${random_password.db_password.result}"

  # DB instance size
  instance_class                      = "db.m5.xlarge"

  # Storage
  storage_type                        = "gp2"
  allocated_storage                   = 100
  max_allocated_storage               = 2000

  # Availability & durability
  multi_az                            = true

  # Connectivity
  db_subnet_group_name                = aws_db_subnet_group.sg.id

  publicly_accessible                 = false
  vpc_security_group_ids              = [aws_security_group.sg.id]
  port                                = var.rds_port

  # Database authentication
  iam_database_authentication_enabled = false 

  # Additional configuration
  parameter_group_name                = "default.postgres12"

  # Backup
  backup_retention_period             = 14
  backup_window                       = "03:00-04:00"
  final_snapshot_identifier           = "gitlab-postgresql-final-snapshot-${random_string.db_suffix.result}" 
  delete_automated_backups            = true
  skip_final_snapshot                 = false

  # Encryption
  storage_encrypted                   = true

  # Maintenance
  auto_minor_version_upgrade          = true
  maintenance_window                  = "Sat:00:00-Sat:02:00"

  # Deletion protection
  deletion_protection                 = false

  tags = {
    Environment = "core"
  }
}

resource "aws_db_instance" "gitlab-replica" {
  count = 2

  # Engine options
  engine                              = "postgres"
  engine_version                      = "12.5"

  # Settings
  name                                = "gitlabhq_production"
  identifier                          = "gitlab-replica-${count.index}"

  replicate_source_db                 = "gitlab-primary"  
  skip_final_snapshot                 = true
  final_snapshot_identifier           = null

  # DB instance size
  instance_class                      = "db.m5.xlarge"

  # Storage
  storage_type                        = "gp2"
  allocated_storage                   = 100
  max_allocated_storage               = 2000

  publicly_accessible                 = false
  vpc_security_group_ids              = [aws_security_group.sg.id]
  port                                = var.rds_port

  # Database authentication
  iam_database_authentication_enabled = false 

  # Additional configuration
  parameter_group_name                = "default.postgres12"

  # Encryption
  storage_encrypted                   = true

  # Deletion protection
  deletion_protection                 = false

  tags = {
    Environment = "core"
  }

  depends_on = [
    aws_db_instance.gitlab-primary
  ]
}

resource "aws_db_subnet_group" "sg" {
  name       = "gitlab"
  subnet_ids = [for s in aws_subnet.private : s.id]

  tags = {
    Environment = "core"
    Name        = "gitlab"
  }
}

resource "aws_security_group" "sg" {
  name        = "gitlab"
  description = "Allow inbound/outbound traffic"
  vpc_id      = aws_vpc.devops.id

  ingress {
    from_port       = var.rds_port
    to_port         = var.rds_port
    protocol        = "tcp"
    cidr_blocks = [for s in aws_subnet.private : s.cidr_block]
  }

  egress {
    from_port       = 0
    to_port         = 65535
    protocol        = "tcp"
    cidr_blocks = [for s in aws_subnet.private : s.cidr_block]
  }

  tags = {
    Environment = "core"
    Name        = "gitlab"
  }
}

Also here, you can encrypt the RDS instance with Amazon KMS [3]

External Redis

Redis persists GitLab job data. We use Amazon Elasticache to create the redis cluster. Only EKS worker nodegroups are authorized to access the redis nodes.

plan/elasticache.tf

resource "aws_elasticache_cluster" "gitlab" {
  cluster_id           = "cluster-gitlab"
  engine               = "redis"
  node_type            = "cache.m4.xlarge"
  num_cache_nodes      = 1
  parameter_group_name = "default.redis3.2"
  engine_version       = "3.2.10"
  port                 = 6379
  subnet_group_name    = aws_elasticache_subnet_group.gitlab.name
  security_group_ids   = [aws_security_group.redis.id]
}

resource "aws_elasticache_subnet_group" "gitlab" {
  name       = "gitlab-cache-subnet"
  subnet_ids = [for s in aws_subnet.private : s.id]
}

resource "aws_security_group" "redis" {
  name        = "ElasticacheRedisSecurityGroup"
  description = "Communication between the redis and eks worker nodegroups"
  vpc_id      = aws_vpc.devops.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "ElasticacheRedisSecurityGroup"
  }
}

resource "aws_security_group_rule" "redis_inbound" {
  description              = "Allow eks nodes to communicate with Redis"
  from_port                = 6379
  protocol                 = "tcp"
  security_group_id        = aws_security_group.redis.id
  source_security_group_id = aws_eks_cluster.devops.vpc_config[0].cluster_security_group_id
  to_port                  = 6379
  type                     = "ingress"
}

External persistent volume

Gitaly persists the Git repositories and requires persistent storage, configured through persistent volumes that specify which disks the cluster has access to.

Gitlab currently recommends using manual provisioning of persistent volumes. Amazon EKS clusters default to spanning multiple zones. Dynamic provisioning, if not configured to use a storage class locked to a particular zone leads to a scenario where pods may exist in a different zone from storage volumes and be unable to access data. [5]

plan/ebs.tf

resource "aws_ebs_volume" "gitaly" {
  availability_zone = var.az[0]
  size              = 200
  type              = "gp2"

  tags = {
    Name = "gitaly"
  }
}

Also here, you can encrypt the EBS resource with Amazon KMS [4]

We have completed creating the necessary AWS resources for Gitlab. We are now starting to create the Kubernetes resources.

Kubernetes resources

In this section we create:

gitlab namespace,
aws-access service account needed to access the S3 object storages,
gitlab-postgres secret to store the db password,
s3-storage-credentials and s3-registry-storage-credentials secrets to access the S3 object storages,
shell-secret for gitlab shell,
The storage class, persistent volume and the persistent volume claim used by gitaly.

plan/k8s.tf

esource "kubernetes_namespace" "gitlab" {
  metadata {
    name = "gitlab"
  }
}

resource "kubernetes_service_account" "gitlab" {
  metadata {
    name      = "aws-access"
    namespace = "gitlab"

    labels = {
      "app.kubernetes.io/name" = "aws-access"
    }
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.gitlab-access.arn
    }
  }
}

resource "kubernetes_secret" "gitlab-postgres" {
  metadata {
    name       = "gitlab-postgres"
    namespace  = "gitlab"
  }

  data = {
   psql-password = "p${random_password.db_password.result}"
  }
}

resource "kubernetes_secret" "s3-storage-credentials" {
  metadata {
    name       = "s3-storage-credentials"
    namespace  = "gitlab"
  }

  data = {
    connection = data.template_file.rails-s3-yaml.rendered
  }
}

data "template_file" "rails-s3-yaml" {
  template = <<EOF
provider: AWS
region: ${var.region}

EOF
}

resource "kubernetes_secret" "s3-registry-storage-credentials" {
  metadata {
    name       = "s3-registry-storage-credentials"
    namespace  = "gitlab"
  }

  data = {
    config = data.template_file.registry-s3-yaml.rendered
  }
}

data "template_file" "registry-s3-yaml" {
  template = <<EOF
s3:
    bucket: ${aws_s3_bucket.gitlab-registry.id}
    region: ${var.region}
    v4auth: true
EOF
}

resource "random_password" "shell-secret" {
  length = 12
  special = true
  upper = true
}

resource "kubernetes_secret" "shell-secret" {
  metadata {
    name       = "shell-secret"
    namespace  = "gitlab"
  }

  data = {
    password = random_password.shell-secret.result
  }
}

resource "kubernetes_persistent_volume" "gitaly" {
  metadata {
    name      = "gitaly-pv"
  }
  spec {
    capacity = {
      storage = "200Gi"
    }
    access_modes = ["ReadWriteOnce"]
    storage_class_name = "ebs-gp2"
    persistent_volume_source {
        aws_elastic_block_store {
            fs_type   = "ext4"
            volume_id = aws_ebs_volume.gitaly.id
        }
    }
  }
}

resource "kubernetes_persistent_volume_claim" "gitaly" {
  metadata {
    name      = "repo-data-gitlab-gitaly-0"
    namespace = "gitlab"
  }
  spec {
    access_modes = ["ReadWriteOnce"]
    storage_class_name = "ebs-gp2"
    resources {
      requests = {
        storage = "200Gi"
      }
    }
    volume_name = kubernetes_persistent_volume.gitaly.metadata.0.name
  }
}

resource "kubernetes_storage_class" "gitaly" {
  metadata {
    name = "ebs-gp2"
  }
  storage_provisioner = "kubernetes.io/aws-ebs"
  reclaim_policy      = "Retain"

  parameters = {
    type = "gp2"
  }

  allowed_topologies {
    match_label_expressions {
      key = "failure-domain.beta.kubernetes.io/zone"
      values = var.az
    }
  }
}

Helm

We finish by configuring the helm chart.

plan/gitlab.tf

data "template_file" "gitlab-values" {
  template = <<EOF

# Values for gitlab/gitlab chart on EKS
global:
  serviceAccount:
    enabled: true
    create: false
    name: aws-access
  platform:
    eksRoleArn: ${aws_iam_role.gitlab-access.arn}

  nodeSelector:
    eks.amazonaws.com/nodegroup: private

  shell:
    authToken:
      secret: ${kubernetes_secret.shell-secret.metadata.0.name}
      key: password

  edition: ce

  hosts:
    domain: ${var.public_dns_name}
    https: true
    gitlab:
      name: gitlab.${var.public_dns_name}
      https: true
    ssh: ~

  ## doc/charts/globals.md#configure-ingress-settings
  ingress:
    tls:
      enabled: false

  ## doc/charts/globals.md#configure-postgresql-settings
  psql:
    password:
       secret: ${kubernetes_secret.gitlab-postgres.metadata.0.name}
       key: psql-password
    host: ${aws_db_instance.gitlab-primary.address}
    port: ${var.rds_port}
    username: gitlab
    database: gitlabhq_production
    load_balancing:
      hosts:
      - ${aws_db_instance.gitlab-replica[0].address}
      - ${aws_db_instance.gitlab-replica[1].address}

  redis:
    password:
      enabled: false
    host: ${aws_elasticache_cluster.gitlab.cache_nodes[0].address}

  ## doc/charts/globals.md#configure-minio-settings
  minio:
    enabled: false

  ## doc/charts/globals.md#configure-appconfig-settings
  ## Rails based portions of this chart share many settings
  appConfig:
    ## doc/charts/globals.md#general-application-settings
    enableUsagePing: false

    ## doc/charts/globals.md#lfs-artifacts-uploads-packages
    backups:
      bucket: ${aws_s3_bucket.gitlab-backups.id}
    lfs:
      bucket: ${aws_s3_bucket.git-lfs.id}
      connection:
        secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
        key: connection
    artifacts:
      bucket: ${aws_s3_bucket.gitlab-artifacts.id}
      connection:
        secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
        key: connection
    uploads:
      bucket: ${aws_s3_bucket.gitlab-uploads.id}
      connection:
        secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
        key: connection
    packages:
      bucket: ${aws_s3_bucket.gitlab-packages.id}
      connection:
        secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
        key: connection
    ## doc/charts/globals.md#pseudonymizer-settings
    pseudonymizer:
      bucket: ${aws_s3_bucket.gitlab-pseudo.id}
      connection:
        secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
        key: connection
nginx-ingress:
  controller:
    config:
        use-forwarded-headers: "true" 
    service:
        annotations:
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
            service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ${var.acm_gitlab_arn}
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
        targetPorts:
            https: http # the ELB will send HTTP to 443

certmanager-issuer:
  email: ${var.certmanager_issuer_email}

prometheus:
  install: false

redis:
  install: false

# https://docs.gitlab.com/ee/ci/runners/#configuring-runners-in-gitlab
gitlab-runner:
  install: false

gitlab:
  gitaly:
    persistence:
      volumeName: ${kubernetes_persistent_volume_claim.gitaly.metadata.0.name}
    nodeSelector:
      topology.kubernetes.io/zone: ${var.az[0]}
  task-runner:
    backups:
      objectStorage:
        backend: s3
        config:
          secret: ${kubernetes_secret.s3-storage-credentials.metadata.0.name}
          key: connection
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role.gitlab-access.arn}
  webservice:
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role.gitlab-access.arn}
  sidekiq:
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role.gitlab-access.arn}
  migrations:
    # Migrations pod must point directly to PostgreSQL primary
    psql:
      password:
        secret: ${kubernetes_secret.gitlab-postgres.metadata.0.name}
        key: psql-password
      host: ${aws_db_instance.gitlab-primary.address}
      port: ${var.rds_port}

postgresql:
  install: false

gitlab-runner:
  install: true
  rbac:
    create: true
  runners:
    locked: false

registry:
  enabled: true
  annotations:
    eks.amazonaws.com/role-arn: aws_iam_role.gitlab-access.arn
  storage:
    secret: ${kubernetes_secret.s3-registry-storage-credentials.metadata.0.name}
    key: config

EOF
}

resource "helm_release" "gitlab" {
  name       = "gitlab"
  namespace  = "gitlab"
  timeout    = 600

  chart      = "gitlab/gitlab"
  version    = "4.11.3" 

  values     = [data.template_file.gitlab-values.rendered]

  depends_on = [
      aws_eks_node_group.private,
      aws_eks_node_group.public,
      aws_db_instance.gitlab-primary,
      aws_db_instance.gitlab-replica,
      aws_elasticache_cluster.gitlab,
      aws_iam_role_policy.gitlab-access,
      kubernetes_namespace.gitlab,
      kubernetes_secret.gitlab-postgres,
      kubernetes_secret.s3-storage-credentials,
      kubernetes_secret.s3-registry-storage-credentials,
      kubernetes_persistent_volume_claim.gitaly
  ]
}

data "kubernetes_service" "gitlab-webservice" {
  metadata {
    name      = "gitlab-nginx-ingress-controller"
    namespace = "gitlab"
  }

  depends_on = [
    helm_release.gitlab
  ]
}

resource "aws_route53_record" "gitlab" {
 zone_id    = data.aws_route53_zone.public.zone_id
 name       = "gitlab.${var.public_dns_name}"
 type       = "CNAME"
 ttl        = "300"
 records    = [data.kubernetes_service.gitlab-webservice.status.0.load_balancer.0.ingress.0.hostname]

 depends_on = [
   helm_release.gitlab,
   data.kubernetes_service.gitlab-webservice
 ]
}

Deployment

We've finished creating our terraform files, let's get ready for deployment!

plan/main.tf

data "aws_caller_identity" "current" {}

data "aws_route53_zone" "public" {
  name = "${var.public_dns_name}."
}

plan/output.tf

output "eks-endpoint" {
    value = aws_eks_cluster.devops.endpoint
}

output "kubeconfig-certificate-authority-data" {
    value = aws_eks_cluster.devops.certificate_authority[0].data
}

output "eks_issuer_url" {
    value = aws_iam_openid_connect_provider.openid.url
}

output "nat1_ip" {
    value = aws_eip.nat["public-devops-1"].public_ip
}

output "nat2_ip" {
    value = aws_eip.nat["public-devops-2"].public_ip
}

plan/variables.tf

variable "region" {
  type = string
}

variable "az" {
  type    = list(string)
  default = ["eu-west-1a", "eu-west-1b"]
}

variable "certmanager_issuer_email" {
  type = string  
}

variable "vpc_cidr_block" {
  type = string
}

variable "eks_cluster_name" {
  type = string
  default = "devops"
}

variable "rds_port" {
  type    = number
  default = 5432
}

variable "acm_gitlab_arn" {
  type = string
}

variable "private_network_config" {
  type = map(object({
      cidr_block               = string
      associated_public_subnet = string
  }))

  default = {
    "private-devops-1" = {
        cidr_block               = "10.0.0.0/23"
        associated_public_subnet = "public-devops-1"
    },
    "private-devops-2" = {
        cidr_block               = "10.0.2.0/23"
        associated_public_subnet = "public-devops-2"
    }
  }
}

locals {
    private_nested_config = flatten([
        for name, config in var.private_network_config : [
            {
                name                     = name
                cidr_block               = config.cidr_block
                associated_public_subnet = config.associated_public_subnet
            }
        ]
    ])
}

variable "public_network_config" {
  type = map(object({
      cidr_block              = string
  }))

  default = {
    "public-devops-1" = {
        cidr_block = "10.0.8.0/23"
    },
    "public-devops-2" = {
        cidr_block = "10.0.10.0/23"
    }
  }
}

locals {
    public_nested_config = flatten([
        for name, config in var.public_network_config : [
            {
                name                    = name
                cidr_block              = config.cidr_block
            }
        ]
    ])
}

variable "public_dns_name" {
  type    = string
}

variable "authorized_source_ranges" {
  type        = string
  description = "Addresses or CIDR blocks which are allowed to connect to the Gitlab IP address. The default behavior is to allow anyone (0.0.0.0/0) access. You should restrict access to external IPs that need to access the Gitlab cluster."
  default     = "0.0.0.0/0"
}

plan/backend.tf

terraform {
  backend "s3" {
  }
}

plan/versions.tf

terraform {
  required_version = ">= 0.13"
}

plan/provider.tf

provider "aws" {
  region = var.region
}

provider "kubernetes" {
  host                   = aws_eks_cluster.devops.endpoint

  cluster_ca_certificate = base64decode(
    aws_eks_cluster.devops.certificate_authority[0].data
  )

  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["eks", "get-token", "--cluster-name", var.eks_cluster_name]
    command     = "aws"
  }
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.devops.endpoint

    cluster_ca_certificate = base64decode(
        aws_eks_cluster.devops.certificate_authority[0].data
    )

    exec {
        api_version = "client.authentication.k8s.io/v1alpha1"
        args        = ["eks", "get-token", "--cluster-name", var.eks_cluster_name]
        command     = "aws"
    }
  }
}

plan/terraform.tfvars

az                       = ["<AWS_REGION>a", "<AWS_REGION>b", "<AWS_REGION>c"]
region                   = "<AWS_REGION>"
acm_gitlab_arn           = "<ACM_GITLAB_ARN>"
vpc_cidr_block           = "10.0.0.0/16"
public_dns_name          = "<PUBLIC_DNS_NAME>"
authorized_source_ranges = "<LOCAL_IP_RANGES>"
certmanager_issuer_email = "<CERTMANAGER_ISSUER_EMAIL>"

Initialize AWS devops infrastructure. The states will be saved in AWS.

export AWS_PROFILE=<MY_PROFILE>
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION=eu-west-1
export EKS_CLUSTER_NAME="devops"
export R53_HOSTED_ZONE_ID=<R53_HOSTED_ZONE_ID>
export ACM_GITLAB_ARN=<ACM_GITLAB_ARN>
export CERTMANAGER_ISSUER_EMAIL=<CERTMANAGER_ISSUER_EMAIL>
export PUBLIC_DNS_NAME=<PUBLIC_DNS_NAME>
export TERRAFORM_BUCKET_NAME=bucket-${AWS_ACCOUNT_ID}-${AWS_REGION}-terraform-backend

# Create bucket
aws s3api create-bucket \
     --bucket $TERRAFORM_BUCKET_NAME \
     --region $AWS_REGION \
     --create-bucket-configuration LocationConstraint=$AWS_REGION

# Make it not public     
aws s3api put-public-access-block \
    --bucket $TERRAFORM_BUCKET_NAME \
    --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Enable versioning
aws s3api put-bucket-versioning \
    --bucket $TERRAFORM_BUCKET_NAME \
    --versioning-configuration Status=Enabled

cd plan
terraform init \
    -backend-config="bucket=$TERRAFORM_BUCKET_NAME" \
    -backend-config="key=devops/gitlab/terraform-state" \
    -backend-config="region=$AWS_REGION"

Complete plan/terraform.tfvars and run

sed -i "s/<LOCAL_IP_RANGES>/$(curl -s http://checkip.amazonaws.com/)\/32/g; s/<PUBLIC_DNS_NAME>/${PUBLIC_DNS_NAME}/g; s/<AWS_ACCOUNT_ID>/${AWS_ACCOUNT_ID}/g; s/<AWS_REGION>/${AWS_REGION}/g; s/<EKS_CLUSTER_NAME>/${EKS_CLUSTER_NAME}/g; s,<ACM_GITLAB_ARN>,${ACM_GITLAB_ARN},g; s/<CERTMANAGER_ISSUER_EMAIL>/${CERTMANAGER_ISSUER_EMAIL}/g;" terraform.tfvars

terraform apply

Access the EKS Cluster using

aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME

To deploy the Amazon EBS CSI driver, run one of the following commands based on your AWS Region:

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

Annotate the ebs-csi-controller-sa Kubernetes service account with the ARN of the IAM role that you created in terraform:

kubectl annotate serviceaccount ebs-csi-controller-sa \
  -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::$AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole

Delete the driver pods:

kubectl delete pods \
  -n kube-system \
  -l=app=ebs-csi-controller

Note: The driver pods are automatically redeployed with the IAM permissions from the IAM policy assigned to the role. For more information, see Amazon EBS CSI driver.

Check if all resources have been created properly:

$ kubectl config set-context --current --namespace=gitlab
$ kubectl get all

NAME                                                        READY   STATUS      RESTARTS   AGE
pod/gitlab-cainjector-67dbdcc896-trv82                      1/1     Running     0          12m
pod/gitlab-cert-manager-69bd6d746f-gllsw                    1/1     Running     0          12m
pod/gitlab-gitaly-0                                         1/1     Running     0          12m
pod/gitlab-gitlab-exporter-7f84659548-pdgg7                 1/1     Running     0          12m
pod/gitlab-gitlab-runner-78f9779cc5-ln8jv                   1/1     Running     3          12m
pod/gitlab-gitlab-shell-79587877cf-k9cms                    1/1     Running     0          12m
pod/gitlab-gitlab-shell-79587877cf-sp7qd                    1/1     Running     0          12m
pod/gitlab-issuer-1-kmmlf                                   0/1     Completed   0          12m
pod/gitlab-migrations-1-kzwbw                               0/1     Completed   0          12m
pod/gitlab-nginx-ingress-controller-d6cfd66cb-pwqw9         1/1     Running     0          12m
pod/gitlab-nginx-ingress-controller-d6cfd66cb-q2vqs         1/1     Running     0          12m
pod/gitlab-nginx-ingress-default-backend-658cc89589-tbn4f   1/1     Running     0          12m
pod/gitlab-registry-86d9c8f9cd-m6f9b                        1/1     Running     0          12m
pod/gitlab-registry-86d9c8f9cd-t99dn                        1/1     Running     0          12m
pod/gitlab-sidekiq-all-in-1-v1-5c5dcd6dbb-k9mpr             1/1     Running     0          12m
pod/gitlab-task-runner-5dff588987-z59k8                     1/1     Running     0          12m
pod/gitlab-webservice-default-79dc48bcd4-8zc7t              2/2     Running     0          12m
pod/gitlab-webservice-default-79dc48bcd4-lq2j9              2/2     Running     0          12m

NAME                                              TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                   AGE
service/gitlab-cert-manager                       ClusterIP      172.20.21.34     <none>                                                                    9402/TCP                                  12m
service/gitlab-gitaly                             ClusterIP      None             <none>                                                                    8075/TCP,9236/TCP                         12m
service/gitlab-gitlab-exporter                    ClusterIP      172.20.183.92    <none>                                                                    9168/TCP                                  12m
service/gitlab-gitlab-shell                       ClusterIP      172.20.207.85    <none>                                                                    22/TCP                                    12m
service/gitlab-nginx-ingress-controller           LoadBalancer   172.20.113.153   ae030366025b247398e8230174fbc4d3-1830148250.eu-west-1.elb.amazonaws.com   80:30698/TCP,443:31788/TCP,22:32155/TCP   12m
service/gitlab-nginx-ingress-controller-metrics   ClusterIP      172.20.82.7      <none>                                                                    9913/TCP                                  12m
service/gitlab-nginx-ingress-default-backend      ClusterIP      172.20.166.82    <none>                                                                    80/TCP                                    12m
service/gitlab-registry                           ClusterIP      172.20.47.110    <none>                                                                    5000/TCP                                  12m
service/gitlab-webservice-default                 ClusterIP      172.20.70.7      <none>                                                                    8080/TCP,8181/TCP                         12m

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gitlab-cainjector                      1/1     1            1           13m
deployment.apps/gitlab-cert-manager                    1/1     1            1           13m
deployment.apps/gitlab-gitlab-exporter                 1/1     1            1           13m
deployment.apps/gitlab-gitlab-runner                   1/1     1            1           13m
deployment.apps/gitlab-gitlab-shell                    2/2     2            2           13m
deployment.apps/gitlab-nginx-ingress-controller        2/2     2            2           13m
deployment.apps/gitlab-nginx-ingress-default-backend   1/1     1            1           13m
deployment.apps/gitlab-registry                        2/2     2            2           13m
deployment.apps/gitlab-sidekiq-all-in-1-v1             1/1     1            1           13m
deployment.apps/gitlab-task-runner                     1/1     1            1           13m
deployment.apps/gitlab-webservice-default              2/2     2            2           13m

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/gitlab-cainjector-67dbdcc896                      1         1         1       13m
replicaset.apps/gitlab-cert-manager-69bd6d746f                    1         1         1       13m
replicaset.apps/gitlab-gitlab-exporter-7f84659548                 1         1         1       13m
replicaset.apps/gitlab-gitlab-runner-78f9779cc5                   1         1         1       13m
replicaset.apps/gitlab-gitlab-shell-79587877cf                    2         2         2       13m
replicaset.apps/gitlab-nginx-ingress-controller-d6cfd66cb         2         2         2       13m
replicaset.apps/gitlab-nginx-ingress-default-backend-658cc89589   1         1         1       13m
replicaset.apps/gitlab-registry-86d9c8f9cd                        2         2         2       13m
replicaset.apps/gitlab-sidekiq-all-in-1-v1-5c5dcd6dbb             1         1         1       13m
replicaset.apps/gitlab-task-runner-5dff588987                     1         1         1       13m
replicaset.apps/gitlab-webservice-default-79dc48bcd4              2         2         2       13m

NAME                             READY   AGE
statefulset.apps/gitlab-gitaly   1/1     13m

NAME                                                             REFERENCE                               TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/gitlab-gitlab-shell          Deployment/gitlab-gitlab-shell          <unknown>/100m   2         10        2          13m
horizontalpodautoscaler.autoscaling/gitlab-registry              Deployment/gitlab-registry              <unknown>/75%    2         10        2          13m
horizontalpodautoscaler.autoscaling/gitlab-sidekiq-all-in-1-v1   Deployment/gitlab-sidekiq-all-in-1-v1   <unknown>/350m   1         10        1          13m
horizontalpodautoscaler.autoscaling/gitlab-webservice-default    Deployment/gitlab-webservice-default    <unknown>/1      2         10        2          13m

NAME                            COMPLETIONS   DURATION   AGE
job.batch/gitlab-issuer-1       1/1           10s        13m
job.batch/gitlab-migrations-1   1/1           3m1s       13m

Connect to the Gitlab Web application with the root user and the initial root password:

$ kubectl get secret gitlab-gitlab-initial-root-password \
     -o go-template='{{.data.password}}' | base64 -d && echo

That's it!

The source code is available on Gitlab.

Conclusion

We discovered in this article how to configure and deploy a production ready Gitlab on Amazon EKS.

Hope you enjoyed reading this blog post.

If you have any questions or feedback, please feel free to leave a comment.

Thanks for reading!

Documentation

[1] https://docs.gitlab.com/charts/advanced/external-object-storage/#docker-registry-images
[2] https://docs.gitlab.com/charts/advanced/external-object-storage/#s3-encryption
[3]
https://docs.aws.amazon.com/kms/latest/developerguide/services-rds.html
[4] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html
[5] https://docs.gitlab.com/charts/installation/cloud/eks.html#persistent-volume-management