Forem: M.M.Monirul Islam

Revolutionizing Monitoring: Exploring the Power of Amazon Managed Service for Prometheus

M.M.Monirul Islam — Wed, 06 Mar 2024 12:38:10 +0000

In the dynamic landscape of cloud computing, efficient monitoring is paramount for ensuring the reliability and performance of applications. Amazon Web Services (AWS) has recently introduced a game-changer in the realm of observability – the Amazon Managed Service for Prometheus. This blog dives into the capabilities, benefits, and transformative impact of this managed service, showcasing how it empowers businesses to scale their monitoring infrastructure seamlessly.

Understanding the Need for Advanced Monitoring

In this section, we explore the evolving needs of modern applications and the challenges associated with traditional monitoring solutions. From the complexities of managing scale to the demand for real-time insights, we set the stage for the emergence of Amazon Managed Service for Prometheus.

Unveiling Amazon Managed Service for Prometheus

Delve into the features and architecture of Amazon Managed Service for Prometheus. Highlight the simplicity of setup, seamless integration with existing AWS services, and the scalability that it offers. Emphasize how this managed service lifts the burden of infrastructure management, allowing teams to focus on deriving actionable insights from their monitoring data.

Key Benefits and Use Cases

Examine the tangible benefits of adopting Amazon Managed Service for Prometheus. From cost savings to improved reliability, discuss how this service enhances observability for applications running on AWS. Provide real-world use cases that showcase the versatility of Prometheus in different scenarios.

Seamless Integration with AWS Ecosystem

Explore the tight integration of Amazon Managed Service for Prometheus with other AWS services, such as Amazon CloudWatch and AWS Identity and Access Management (IAM). Discuss how this interoperability enriches the monitoring experience and contributes to a holistic approach to application observability.

Getting Started - A Step-by-Step Guide

Offer readers a practical guide on getting started with Amazon Managed Service for Prometheus. Cover the essential steps, share best practices, and highlight any considerations for a smooth onboarding process.

Step 1: Accessing the AWS Management Console
Begin by logging into the AWS Management Console. Navigate to the Amazon Managed Service for Prometheus section and initiate the setup process.

Step 2: Creating a Prometheus Workspace
Follow the prompts to create a new Prometheus workspace. Define the required parameters such as the workspace name, desired region, and any specific configurations based on your monitoring needs.

arn:aws:aps:us-east-1:726459634338:workspace/ws-420cc8c0-93ff-4bfd-9dfb-48e0e9a02d5a

Endpoint - remote write URL
https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-420cc8c0-93ff-4bfd-9dfb-48e0e9a02d5a/api/v1/remote_write

Endpoint - query URL
https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-420cc8c0-93ff-4bfd-9dfb-48e0e9a02d5a/api/v1/query

Step 3: Set up IAM roles for service accounts
For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running. These roles are also called service roles.

With service roles, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see IAM roles for service accounts.

If you have not already set up these roles, follow this instructions to set up the roles: Set up service roles for the ingestion of metrics from Amazon EKS clusters.

Please make sure you have created an IAM role called amp-iamproxy-ingest-role before continuing.

Step 4: Upgrade your existing Prometheus server using Helm (for Prometheus version 2.26.0 or later)

The instructions in this section includes setting up remote write and sigv4 to authenticate and authorize the Prometheus server to remote write to your AMP workspace.

To set up remote write from a Prometheus server using Helm chart:
Create a new remote write section in your Helm configuration file:

serviceAccounts:
    server:
        name: "amp-iamproxy-ingest-service-account"
        annotations:
            eks.amazonaws.com/role-arn: "arn:aws:iam::726459634338:role/amp-iamproxy-ingest-role"
server:
    remoteWrite:
        - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-420cc8c0-93ff-4bfd-9dfb-48e0e9a02d5a/api/v1/remote_write
          sigv4:
            region: us-east-1
          queue_config:
            max_samples_per_send: 1000
            max_shards: 200
            capacity: 2500

Update your existing Prometheus Server configuration using Helm:

First, find your Helm chart name by entering the following command:

helm ls --all-namespaces

In the output from this command, look for a chart with a name that includes "prometheus".

Then enter the following command:

helm upgrade prometheus-chart-name prometheus-community/prometheus -n prometheus_namespace -f my_prometheus_values_yaml --version current_helm_chart_version

Replace these placeholders with the following:

Replace prometheus-chart-name with the name of the helm chart you found with the previous command.
Replace prometheus_namespace with the name of the Kubernetes namespace where your Prometheus Server is installed.
Replace my_prometheus_values_yaml with the path to your Helm configuration file.
Replace current_helm_chart_version with the current version of the Prometheus Server Helm chart you found with the previous command.

Grafana

Step-1: Follow the following procedure to setting up managed grafana.

Step 2: Configuring Data Retention and Storage Options
Fine-tune your monitoring environment by configuring data retention policies and storage options. Tailor these settings to align with your application's requirements and expected data volumes.

Step 3: Integrating with AWS Resources
Connect Amazon Managed Service for Prometheus with your existing AWS resources. This could include linking it to Amazon Elastic Kubernetes Service (EKS) clusters, Amazon EC2 instances, or any other AWS services generating monitoring data.

Step 4: Setting Up Alerts and Notifications
Enhance your observability by configuring alerts and notifications. Define thresholds for metrics that matter most to your applications and establish alerting mechanisms to keep your team informed of any deviations from expected behavior.

Step 5: Exploring Querying Capabilities
Dive into the querying capabilities of Amazon Managed Service for Prometheus. Learn how to write PromQL queries to extract valuable insights from your monitoring data. Leverage this functionality to troubleshoot issues, analyze trends, and optimize performance.

Step 6: Integrating with AWS CloudWatch
Explore the seamless integration between Amazon Managed Service for Prometheus and AWS CloudWatch. Understand how these services work together to provide a comprehensive monitoring solution for your AWS environment.

Step 7: Scaling and Optimization
As your application grows, ensure that your monitoring solution scales accordingly. Learn best practices for scaling Amazon Managed Service for Prometheus and optimizing its performance to meet the evolving demands of your applications.

Wrap up the blog by summarizing the key takeaways and emphasizing the transformative potential of Amazon Managed Service for Prometheus. Encourage readers to explore this powerful monitoring solution to elevate their observability practices and stay ahead in the ever-evolving landscape of cloud-native applications.

By shedding light on the capabilities of Amazon Managed Service for Prometheus, this blog aims to guide organizations toward a more efficient and scalable approach to monitoring, ultimately enhancing their ability to deliver reliable and performant applications on AWS.

Kong Gateway on AWS EKS: A Journey into Cloud-native API Management

M.M.Monirul Islam — Tue, 16 Jan 2024 04:32:39 +0000

In the ever-evolving landscape of cloud-native applications, managing APIs efficiently is crucial. An API Gateway plays a pivotal role as a bridge between clients and backends, orchestrating requests and responses while handling a myriad of tasks such as CORS validation, TLS termination, authentication, and more. In this blog post, we embark on a journey into cloud-native API management with Kong Gateway deployed on an AWS EKS cluster.

Understanding the Role of an API Gateway

At its core, an API Gateway acts as a proxy, mediating communication between clients and backend services. It streamlines the process by handling various tasks in transit, including CORS validation, TLS termination, JWT authentication, header injection, session management, response transformation, rate-limiting, ACLs, and much more. This intermediary layer ensures seamless and secure interactions within a microservices architecture.

Introducing Kong Gateway

Developed by KongHQ, Kong Gateway stands out as a lightweight and decentralized API Gateway solution. Operating as a Lua application within NGINX and distributed with OpenResty, Kong Gateway sets the stage for modular extensibility through a rich ecosystem of plugins. Whether your API management needs are basic or complex, Kong Gateway provides a scalable and versatile solution.

The Evolution: DBLess Kong Gateway

Traditionally, Kong Gateway configurations, including routes, services, and plugins, were stored in a database. However, the landscape shifted with the advent of "DBLess" Kong Gateway, also known as the "declarative" method. In this mode, configuration management shifts entirely to code, typically saved as a declarative.yaml file. This paradigm shift brings about several advantages:

1. Version Control:
Configuration becomes easily versionable, enabling seamless collaboration and tracking changes over time.
2. Simplicity and Agility:
Eliminating the need for a separate database streamlines deployment and enhances agility in managing configurations.
3. Infrastructure as Code (IaC):
The move towards a code-centric approach aligns with the principles of Infrastructure as Code, promoting consistency and reproducibility.
4. Maintenance Ease:
With configurations stored as code, the need for maintaining a separate database diminishes, simplifying the overall maintenance process.

Deploying Kong Gateway on AWS EKS

Now that we grasp the significance of Kong Gateway and the advantages of the DBLess approach, let's delve into the process of deploying Kong Gateway on an AWS EKS cluster.

Declarative Configuration (prd/declarative.yaml): Specifies the Kong services, routes, and associated plugins using a DBLess approach.

_format_version: "2.1"
_transform: true

services:
  - name: invoicing-matrics
    url: https://prdawsinvoicing.example.com/actuator/
    routes:
      - paths:
          - /metrics
        methods:
          - GET
          - POST
          - PUT
        strip_path: true
        protocols:
          - https
          - http
        hosts:
          - k-prd.devopsmonirul.tech
  - name: businessCatalog
    url: https://prdawscatalogos.example.com/catalogs/v1.0/catalogs/catalogs/businessCatalog
    routes:
      - paths:
          - /catalogs/businessCatalog
        methods:
          - GET
          - POST
          - PUT
        strip_path: true
        protocols:
          - https
          - http
        hosts:
          - k-prd.devopsmonirul.tech
  - name: usermanagment
    url: https://prdawsusermanagment.example.com/usermanagement/v1.0/users
    routes:
      - paths:
          - /usermanagement
        methods:
          - GET
          - POST
          - PUT
        strip_path: true
        protocols:
          - https
          - http
        hosts:
          - k-prd.devopsmonirul.tech
  - name: ocr
    url: https://prdawsocr.example.com/ocr/v1.0/actuator
    routes:
      - paths:
          - /ocr
        methods:
          - GET
          - POST
          - PUT
        strip_path: true
        protocols:
          - https
          - http
        hosts:
          - k-prd.devopsmonirul.tech
  #------------------------------------------------- CORS Plugin ----------------------------------------------
plugins:
  - name: prometheus
  - name: cors
    service: invoicing-matrics
    config:
      origins:
        - https://controller-ai.com
        - https://www.example.com
        - https://monirul.digital
        - https://example.com
        - http://local.devopsmonirul.com:4200
        - "*"
      methods:
        - GET
        - POST
        - PUT
      credentials: false
      max_age: 3600
      preflight_continue: false
  - name: cors
    service: businessCatalog
    config:
      origins:
        - "*"
      methods:
        - GET
        - POST
      credentials: false
      max_age: 3600
      preflight_continue: false
  - name: cors
    service: usermanagment
    config:
      origins:
        - "*"
      methods:
        - GET
        - POST
        - PUT
      credentials: false
      max_age: 3600
      preflight_continue: false
  - name: cors
    service: ocr
    config:
      origins:
        - "*"
      methods:
        - GET
        - POST
        - PUT
      credentials: false
      max_age: 3600
      preflight_continue: false

Helm Configuration (prd/kong.yaml):
Helm chart configurations for Kong deployment, including resource limits, ingress controller settings, environment variables, and autoscaling parameters.

deployment:
  serviceAccount:
    create: false

podAnnotations:
  "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

resources:
  limits:
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 1Gi

ingressController:
  enabled: false
  installCRDs: false

env:
  database: "off"
  nginx_worker_processes: "2"
  proxy_access_log: /dev/stdout json_analytics
  proxy_error_log: /dev/stdout
  log_level: "error"
  trusted_ips: "0.0.0.0/0,::/0"
  headers: "off"
  anonymous_reports: "off"
  admin_listen: "off"
  status_listen: 0.0.0.0:8100
  nginx_http_log_format: |
    json_analytics escape=json '{"msec": "$msec", "status": "$status", "request_uri": "$request_uri", "geoip_country_code": "$http_x_client_region", "client_subdivision": "$http_x_client_subdivision", "client_city": "$http_x_client_city","client_city_latlong": "$http_x_client_citylatlong", "connection": "$connection", "connection_requests": "$connection_requests", "pid": "$pid", "request_id": "$request_id", "request_length": "$request_length", "remote_addr": "$remote_addr", "remote_user": "$remote_user", "remote_port": "$remote_port", "time_local": "$time_local", "time_iso8601": "$time_iso8601", "request": "$request", "args": "$args", "body_bytes_sent": "$body_bytes_sent", "bytes_sent": "$bytes_sent", "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "http_x_forwarded_for": "$http_x_forwarded_for", "http_host": "$http_host", "server_name": "$server_name", "request_time": "$request_time", "upstream": "$upstream_addr", "upstream_connect_time": "$upstream_connect_time", "upstream_header_time": "$upstream_header_time", "upstream_response_time": "$upstream_response_time", "upstream_response_length": "$upstream_response_length", "upstream_cache_status": "$upstream_cache_status", "ssl_protocol": "$ssl_protocol", "ssl_cipher": "$ssl_cipher", "scheme": "$scheme", "request_method": "$request_method", "server_protocol": "$server_protocol", "pipe": "$pipe", "gzip_ratio": "$gzip_ratio", "http_cf_ray": "$http_cf_ray", "trace_id": "$http_x_b3_traceid", "proxy_host": "$proxy_host"}'

admin:
  enabled: true
  http:
    enabled: false
  tls:
    enabled: true

proxy:
  type: NodePort
  tls:
    enabled: false

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 11
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/os
              operator: In
              values:
              - linux
            - key: kubernetes.io/arch
              operator: In
              values:
              - amd64
              - arm64
            - key: eks.amazonaws.com/compute-type
              operator: NotIn
              values:
              - fargate

Vertical Pod Autoscaler Configuration (prd/vpa.yaml): Configuration for the Vertical Pod Autoscaler, which adjusts resource requests and limits for pods based on their usage.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: prd-kong-vpa
  namespace: prd-kong
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: prd-kong
  updatePolicy:
    updateMode: "Off"

Ingress Configuration (prd/ingress.yaml): Configuration for the Ingress resource, specifying rules and annotations for AWS ALB.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prd-kong-ingress
  namespace: prd-kong
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
        { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:234366607644:certificate/c25d25f3-78ae-4197-a806-1882f6b947dc
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/success-codes: 200,404,301,302
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: k-prd.devopsmonirul.tech
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: prd-kong-proxy
                port:
                  number: 80

Common Scripts (scripts/common.sh): Shell script with functions for common tasks, such as printing colored text, checking the existence of commands, and defining an exit strategy.

#!/usr/bin/env bash

set -euo pipefail

export HELM_VERSION="v3.7.2"

function red() {
    local text="${1:- }"
    echo -e "\033[31m${text}\033[0m"
}

function green() {
    local text="${1:- }"
    echo -e "\033[32m${text}\033[0m"
}

function yellow() {
    local text="${1:- }"
    echo -e "\033[33m${text}\033[0m"
}

function blue() {
    local text="${1:- }"
    echo -e "\033[36m${text}\033[0m"
}

function exists() {
    command -v "${1:-}" > /dev/null 2>&1
}

function graceful_exit() {
    red "${1:- }"
    exit 1
}

function bastion_command() {
    local command="${1:- }"
    if [[ -z "${command}" ]]; then
        error_exit "Command can't be empty."
    else
        gcloud compute ssh --project "${GCP_PROJECT_ID}" --zone "${GCP_ZONE_ID}" "${GCP_USER}@${GCP_SERVER_NAME}" --tunnel-through-iap --command="${1}"
    fi
}

function install_kubectl() {
    if exists /usr/local/bin/kubectl; then
        echo "Kubectl is installed already"
    else
        curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.23.13/2022-10-31/bin/linux/amd64/kubectl
        chmod +x ./kubectl
        mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
        cp ./kubectl /usr/local/bin/kubectl
        kubectl version --short --client
    fi
}

function install_helm() {
    if exists /usr/local/bin/helm; then
        echo "Helm is installed already"
    else
        curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash -s -- --version "${HELM_VERSION}"
    fi
}

function setup_aws_auth() {
    aws eks update-kubeconfig --name "${AWS_PROJECT_ID}" --region "${AWS_LOCATION}" --profile "${CLUSTER_NAME}"
}

Setup Kong Script (scripts/setup-kong.sh): Script for installing kubectl, Helm, adding the Kong Helm repo, setting up AWS authentication, creating namespaces, applying VPA configuration, and deploying Kong using Helm.

#!/usr/bin/env bash

set -euo pipefail

cd "$(dirname "$0")/.."

source scripts/common.sh

green "Installing Kubectl"
install_kubectl

green "Installing helm version => ${HELM_VERSION}"
install_helm

green "Setting up Kong Helm Repo"
helm repo add kong https://charts.konghq.com
helm repo update

green "Setting up AWS Auth"
setup_aws_auth

green "Creating Namespace => ${KONG_NAME}-kong"
kubectl create namespace "${KONG_NAME}"-kong --dry-run=client -o yaml | kubectl apply -f -

green "Set the current namespace"
kubectl config set-context --current --namespace=${KONG_NAME}-kong

green "Setting up VPA Config => ${KONG_NAME}-kong-vpa"
kubectl apply -f ${KONG_NAME}/vpa.yaml || true

if [[ "${KONG_NAME}" == prd ]]; then
    green "Setting up Ingress => ${KONG_NAME}-kong-ingress"
    kubectl apply -f ${KONG_NAME}/ingress.yaml
fi

green "Setting up Kong => ${KONG_NAME}"
helm upgrade \
    --install \
    "${KONG_NAME}" \
    kong/kong \
    --namespace "${KONG_NAME}"-kong \
    --create-namespace \
    -f "${KONG_NAME}"/kong.yaml \
    --set-file dblessConfig.config="${KONG_NAME}"/declarative.yaml \
    --version 2.6.3 \
    --wait \
    --debug

Validate Kong Script (scripts/validate-kong.sh): Script for validating the setup, including Helm diff, VPA configuration, and Ingress configuration (for prd environment).

#!/usr/bin/env bash

set -euo pipefail

cd "$(dirname "$0")/.."

source scripts/common.sh

green "Installing Kubectl"
install_kubectl

green "Installing helm version => ${HELM_VERSION}"
install_helm

green "Setting up Kong Helm Repo"
helm repo add kong https://charts.konghq.com
helm repo update

green "Installing Helm Diff Plugin"
helm plugin install https://github.com/databus23/helm-diff || true

green "Setting up AWS Auth"
setup_aws_auth

green "Set the current namespace"
kubectl config set-context --current --namespace=${KONG_NAME}-kong

green "Validating VPA Config => ${KONG_NAME}-kong-vpa"
kubectl diff -f ${KONG_NAME}/vpa.yaml || true

if [[ "${KONG_NAME}" == prd ]]; then
    green "Validating Ingress => ${KONG_NAME}-kong-ingress"
    kubectl diff -f ${KONG_NAME}/ingress.yaml || true
fi

green "Validating Kong => ${KONG_NAME}"
helm diff upgrade \
    --install \
    "${KONG_NAME}" \
    kong/kong \
    --namespace "${NAMESPACE}" \
    -f ${KONG_NAME}/kong.yaml \
    --set-file dblessConfig.config=${KONG_NAME}/declarative.yaml \
    --version 2.6.3

GitLab CI Configuration (.gitlab-ci.yml): CI/CD pipeline configuration for validating and deploying Kong in the prd environment.

For validations job,

validate-kong-prd:
  stage: validate-kong-prd
  environment: kong-prd
  variables:
    ENV: prd
    KONG_NAME: prd
    AWS_PROJECT_ID: monirul-digital
    CLUSTER_NAME: monirul
    AWS_LOCATION: eu-west-1
    NAMESPACE: ${KONG_NAME}-kong
  extends: [ .common-dependencies ]
  script:
    - aws eks update-kubeconfig --name $AWS_PROJECT_ID --region $AWS_LOCATION --profile $CLUSTER_NAME
    - ./scripts/validate-kong.sh
  only:
    refs:
      - master
      - merge_requests
    changes:
      - prd/**/*
  allow_failure: false

For Deployment Job,

deploy-helm-kong-prd:
  stage: deploy-kong-prd
  environment: kong-prd
  variables:
    ENV: prd
    KONG_NAME: prd
    AWS_PROJECT_ID: monirul-digital
    CLUSTER_NAME: monirul
    AWS_LOCATION: eu-west-1
    NAMESPACE: ${KONG_NAME}-kong
  extends: [ .common-dependencies ]
  script:
    - aws eks update-kubeconfig --name $AWS_PROJECT_ID --region $AWS_LOCATION --profile $CLUSTER_NAME
    - ./scripts/setup-kong.sh
  only:
    refs:
      - master
    changes:
      - prd/**/*
  allow_failure: false
  when: manual

The deployment setup appears to be well-organized and follows best practices for deploying Kong on AWS EKS. The use of GitLab CI/CD enhances automation and ensures consistent deployments.

Next Steps:

Ensure that your deployment scripts and configurations align with your specific requirements and AWS EKS environment.
Monitor Kong Gateway's performance, logs, and metrics in the AWS EKS cluster to identify and address any issues.
Consider further optimizations or enhancements based on specific use cases or evolving requirements.

If you have any specific concerns or questions, feel free to ask!

Connecting Multiple GKE Clusters to a Single Cloud SQL Instance

M.M.Monirul Islam — Tue, 22 Aug 2023 05:03:54 +0000

In a cloud-native environment, managing multiple Google Kubernetes Engine (GKE) clusters is common, especially when different parts of an application need to be isolated. However, these isolated clusters might still need to access a central database like Google Cloud SQL (PostgreSQL). In this blog post, we'll guide you through the process of connecting multiple GKE VPC-native clusters to a single Cloud SQL instance, even when they reside in separate VPC networks.

The Challenge
Imagine you have several GKE clusters, each operating in its own Virtual Private Cloud (VPC) network, for reasons of security and isolation. On the other hand, you have a Cloud SQL instance (PostgreSQL) that serves as the central repository for your application's data. The challenge is to enable seamless connectivity between any of the GKE clusters and the shared Cloud SQL instance, maintaining both network isolation and data security.

The Solution
To address this challenge, we will outline the steps to establish a secure and efficient connection between your GKE VPC-native clusters and the Cloud SQL database, regardless of the VPC network they are in.

Dedicated Node Pools
Begin by creating dedicated node pools within each GKE cluster. These node pools are optimized for backend workloads and ensure that your applications requiring database access have the necessary resources.
Provisioning Cloud SQL
Set up a PostgreSQL database instance in Google Cloud SQL. This instance will serve as the central database for your application's data.

We have many options for creating this instance: we can use Infrastructure as Code (IAC)/manual steps or gcloud command steps provided below:



gcloud sql instances create postgres-instance \
    --project=monirul-test \
    --database-version=POSTGRES_13 \
    --cpu=2 \
    --memory=4GiB \
    --root-password="DummyPass765#" \
    --availability-type=zonal \
    --zone=us-west2-a

Configuring Service Accounts After creating the service account, you need to assign the Cloud SQL Client role to it. To allow this service account to authenticate and access Cloud SQL, you need to generate a JSON key file.

Procedure to creating the service-account-key Secret from a file:



kubectl create secret generic service-account-key --from-file=key.json=/usr/local/monirul-test-c7a15664f347.json

This command will generate a key file named key.json in your current directory. This key file will be used by the Cloud SQL Proxy and your GKE application to authenticate.

Deploying the Cloud SQL Proxy

Deploying the Cloud SQL Proxy as a sidecar is a crucial step in connecting your GKE application to a Cloud SQL database securely. Below is a Kubernetes Deployment YAML manifest demonstrating how to deploy the Cloud SQL Proxy container as a sidecar alongside your main application container:



apiVersion: apps/v1
kind: Deployment
metadata:
    name: test-app-deployment
    annotations:
        linkerd.io/inject: enabled
spec:
    replicas: 1
    strategy:
        type: RollingUpdate
        rollingUpdate:
            maxSurge: 50%
            maxUnavailable: 0
    selector:
        matchLabels:
            app: test-app
    template:
        metadata:
            labels:
                app: test-app
        spec:
            affinity:
                nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        nodeSelectorTerms:
                        - matchExpressions:
                            - key: dedicated
                              operator: In
                              values:
                              - backend
            tolerations:
                - effect: NoSchedule
                  key: dedicated
                  operator: Equal
                  value: backend
            containers:
                - name: test-app
                  image: asia.gcr.io/monirul-test/test-app-deployment:test-app-1.0.0
                  ports:
                      - containerPort: 5000
                        name: test-app
                  resources:
                      limits:
                          cpu: '1'
                          memory: '3Gi'
                      requests:
                          cpu: '0.5'
                          memory: '2Gi'
                  env:
                      - name: DB_PASSWORD
                        valueFrom:
                            secretKeyRef:
                                name: test-app-secret
                                key: DB_PASSWORD
                      - name: APP_PROJECT_ID
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: APP_PROJECT_ID
                      - name: DB_HOST
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: DB_HOST
                      - name: DB_USER
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: DB_USER
                      - name: APP_PORT
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: APP_PORT
                      - name: DB_NAME
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: DB_NAME
                      - name: DB_PORT
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: DB_PORT
                      - name: DATABASE_MAX_CONNECTIONS
                        valueFrom:
                            configMapKeyRef:
                                name: test-app-config
                                key: DATABASE_MAX_CONNECTIONS
                - name: cloud-sql-proxy
                  image: gcr.io/cloudsql-docker/gce-proxy:1.22.0
                  command:
                      - '/cloud_sql_proxy'
                      - '-ip_address_types=PUBLIC'
                      - '-instances=monirul-test:us-west2:postgres-instance=tcp:0.0.0.0:5432'
                      - '-credential_file=/var/secrets/cloud-sql/key.json'
                  resources:
                      requests:
                          cpu: '100m'
                          memory: '128Mi'
                      limits:
                          cpu: '200m'
                          memory: '256Mi'
                  volumeMounts:  # Mount the Secret as a volume
                      - name: service-account-key
                        mountPath: /var/secrets/cloud-sql
            volumes:  # Define the volume that references the Secret
                - name: service-account-key
                  secret:
                      secretName: service-account-key

This YAML manifest defines a Deployment for your application ("test-app-deployment") and includes a sidecar container named "cloud-sql-proxy." The Cloud SQL Proxy handles authentication and encryption for secure connections between your GKE application and the Cloud SQL database.

Ensure you replace the image with your actual image. Once deployed, this configuration allows your GKE application to securely access the Cloud SQL database using the Cloud SQL Proxy.

ConfigMap:



apiVersion: v1
kind: ConfigMap
metadata:
    name: test-app-config
data:
    APP_PORT: '5000'
    APP_PROJECT_ID: monirul-test
    DB_NAME: postgres
    DB_HOST: localhost
    DB_USER: postgres
    DATABASE_MAX_CONNECTIONS: '15'
    DB_PORT: '5432'

Secret:



apiVersion: v1
data:
  DB_PASSWORD: RXhDZXJlODc0Iw==
kind: Secret
metadata:
  name: test-app-secret
  namespace: test-app
type: Opaque

Service.yaml



apiVersion: v1
kind: Service
metadata:
  name: test-app-svc
spec:
  ports:
    - name: test-app-svc-port
      protocol: TCP
      port: 5000
      targetPort: 5000

The provided configurations are clear and should work for your use case. Just make sure that the actual values you use for your application match the ones you've configured in the ConfigMap and Secret.

Testing and Validation Certainly, testing and validation are essential steps to ensure the GKE to Cloud SQL connection works correctly. Your provided commands for testing and validation look good:



$ k exec -it test-app-deployment-786d4bdd56-2sktd -- sh
Defaulted container "test-app" out of: test-app, cloud-sql-proxy
# curl http://localhost:5000
{"message":"hello world"}
# curl http://localhost:5000/data
[["+60102098121","Monirul","Islam","devops.monirul@gmail.com"]]

These commands will help you test various aspects of your setup, including basic connectivity and data retrieval. Be sure to conduct thorough testing, including edge cases and error scenarios, to ensure the reliability and performance of your GKE to Cloud SQL connection.

The Benefits
Connecting multiple GKE VPC-native clusters to a centralized Cloud SQL instance provides several advantages:

Isolation: Each GKE cluster remains isolated within its own VPC network, enhancing security and separation of workloads.

Centralized Data Management: Data consistency is ensured with a single Cloud SQL database instance, simplifying data management.

Scalability: This architecture can scale horizontally to handle increased workloads and data storage requirements.

Security: Strong access controls, encryption, and secure connections guarantee data confidentiality and integrity.

Efficiency: The Cloud SQL Proxy streamlines database connections, reducing latency and ensuring reliable access.

Conclusion
Connecting multiple GKE VPC-native clusters to a shared Cloud SQL instance is a critical step in building scalable, secure, and efficient cloud-native applications. With the right architecture, tools, and best practices, you can seamlessly manage your data while maintaining network isolation and data security.

Reference:
https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine

EKS cluster Monitoring for AWS Fargate with Prometheus and managed Grafana

M.M.Monirul Islam — Wed, 22 Mar 2023 10:33:34 +0000

Firstly, We need to create node group in our existing EKS cluster as metrics are inaccessible to Fargate.

Architecture:

Node group for prometheus:

I actually used IAC (terrafrom) to create eks node group (worker node) for prometheus.



resource "aws_eks_node_group" "monirul_ec2" {
  cluster_name    = aws_eks_cluster.monirul.name
  node_group_name = "monirul_ec2_prometheus"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids = [
    var.private_subnet_id_a,
    var.private_subnet_id_b
  ]

  scaling_config {
    desired_size = 2
    max_size     = 5
    min_size     = 1
  }

  ami_type       = "AL2_x86_64" # AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM
  capacity_type  = "ON_DEMAND"  # ON_DEMAND, SPOT
  disk_size      = 20
  instance_types = ["m5.large"]

  depends_on = [
    aws_iam_role_policy_attachment.node_AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node_AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node_AmazonEC2ContainerRegistryReadOnly,
  ]
}

# EKS Node IAM Role
resource "aws_iam_role" "node" {
  name = "Ec2-Worker-Role"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.node.name
}

Or we can create manually. Go to AWS Management Console -> EKS -> Your cluster -> Compute -> Add node group.

We know that, We have to use EC2 for Prometheus, since will need volumes mounted to it.

While creating node group, we have to attach an IAM role to EC2 worker nodes. For easy demonstration, I have created a new IAM role and attached policies as below.

Run the following command to confirm that your EC2 worker nodes are running properly (2 pods are should be running state).



 $ k get po -n kube-system | grep aws-node
aws-node-jx8dh                                 1/1     Running   0          13d
aws-node-mx4gq                                 1/1     Running   0          13d

Note: Node exporter runs as a daemon set and is responsible for collecting metrics of the host it runs on. Most of these metrics are low-level operating system metrics like vCPU, memory, network, disk (of the host machine, not containers), and hardware statistics, etc. These metrics are inaccessible to Fargate customers since AWS is responsible for the health of the host machine.

Install EBS CSI driver

Prometheus and Grafana needs persistent storage attached to them, which is also called PV(Persistent Volume) in terms of Kubernetes.

For stateful workloads to use Amazon EBS volumes as PV, we have to add aws-ebs-csi-driver into the cluster.

Associating IAM role to Service account

Before we add aws-ebs-csi-driver, we need to create an IAM role, and associate it with Kubernetes service account.

Let's use an example policy file, which you can download using the command below.



curl -sSL -o ebs-csi-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/docs/example-iam-policy.json

Now let's create a new IAM policy with that file.



export EBS_CSI_POLICY_NAME=AmazonEBSCSIPolicy
export AWS_REGION="eu-west-1"
aws iam create-policy \
--region $AWS_REGION \
--policy-name $EBS_CSI_POLICY_NAME \
--policy-document file://ebs-csi-policy.json

export EBS_CSI_POLICY_ARN=$(aws --region eu-west-1 iam list-policies --query 'Policies[?PolicyName==`'$EBS_CSI_POLICY_NAME'`].Arn' --output text)

echo $EBS_CSI_POLICY_ARN
# arn:aws:iam::2343123456678:policy/AmazonEBSCSIPolicy

After that, let's attach the new policy to Kubernetes service account.



eksctl create iamserviceaccount \
  --cluster $EKS_CLUSTER_NAME \
  --name ebs-csi-controller-irsa \
  --namespace kube-system \
  --attach-policy-arn $EBS_CSI_POLICY_ARN \
  --override-existing-serviceaccounts --approve

And now, we're ready to install aws-ebs-csi-driver!

Setting up aws-ebs-csi-driver Helm Repo

Assuming that helm is installed, let's add new helm repository as below.



helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

After adding new helm repository, let's install aws-ebs-csi-driver with below command using helm.



helm upgrade --install aws-ebs-csi-driver \
  --version=1.2.4 \
  --namespace kube-system \
  --set serviceAccount.controller.create=false \
  --set serviceAccount.snapshot.create=false \
  --set enableVolumeScheduling=true \
  --set enableVolumeResizing=true \
  --set enableVolumeSnapshot=true \
  --set serviceAccount.snapshot.name=ebs-csi-controller-irsa \
  --set serviceAccount.controller.name=ebs-csi-controller-irsa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

Creating Namespace => prometheus



kubectl create namespace prometheus --dry-run=client -o yaml | kubectl apply -f -

Setting up Prometheus Helm Repositories



helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Setting up Prometheus



CHART_VERSION="19.7.2"
helm upgrade \
    --install \
    --wait \
    prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --create-namespace \
    --version "${CHART_VERSION}" \
    -f prometheus/prometheus-values.yaml \
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2" \
    --debug

Verify that Prometheus pods are running:



$ kubectl get pods --namespace prometheus
NAME                                                READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-0                           1/1     Running   0          12d
prometheus-kube-state-metrics-6fcf5978bf-dssx2      1/1     Running   0          12d
prometheus-prometheus-node-exporter-677lp           1/1     Running   0          13d
prometheus-prometheus-node-exporter-mwn7j           1/1     Running   0          13d
prometheus-prometheus-pushgateway-fdb75d75f-5pfdt   1/1     Running   0          12d
prometheus-server-5d957cfd5f-thcvn                  2/2     Running   0          12d

Here is the prometheus value that we can use during installation.

prometheus-values.yaml



rbac:
  create: true

podSecurityPolicy:
  enabled: false

imagePullSecrets: []
# - name: "image-pull-secret"

## Define serviceAccount names for components. Defaults to component's fully qualified name.
##
serviceAccounts:
  server:
    create: true
    name: ""
    annotations: {}

## Monitors ConfigMap changes and POSTs to a URL
## Ref: https://github.com/jimmidyson/configmap-reload
##
configmapReload:
  prometheus:
    ## If false, the configmap-reload container will not be deployed
    ##
    enabled: true

    ## configmap-reload container name
    ##
    name: configmap-reload

    ## configmap-reload container image
    ##
    image:
      repository: jimmidyson/configmap-reload
      tag: v0.8.0
      # When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value).
      digest: ""
      pullPolicy: IfNotPresent

    # containerPort: 9533

    ## Additional configmap-reload container arguments
    ##
    extraArgs: {}
    ## Additional configmap-reload volume directories
    ##
    extraVolumeDirs: []


    ## Additional configmap-reload mounts
    ##
    extraConfigmapMounts: []
      # - name: prometheus-alerts
      #   mountPath: /etc/alerts.d
      #   subPath: ""
      #   configMap: prometheus-alerts
      #   readOnly: true

    ## Security context to be added to configmap-reload container
    containerSecurityContext: {}

    ## configmap-reload resource requests and limits
    ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
    ##
    resources: {}

server:
  ## Prometheus server container name
  ##
  name: server

  ## Use a ClusterRole (and ClusterRoleBinding)
  ## - If set to false - we define a RoleBinding in the defined namespaces ONLY
  ##
  ## NB: because we need a Role with nonResourceURL's ("/metrics") - you must get someone with Cluster-admin privileges to define this role for you, before running with this setting enabled.
  ##     This makes prometheus work - for users who do not have ClusterAdmin privs, but wants prometheus to operate on their own namespaces, instead of clusterwide.
  ##
  ## You MUST also set namespaces to the ones you have access to and want monitored by Prometheus.
  ##
  # useExistingClusterRoleName: nameofclusterrole

  ## namespaces to monitor (instead of monitoring all - clusterwide). Needed if you want to run without Cluster-admin privileges.
  # namespaces:
  #   - yournamespace

  # sidecarContainers - add more containers to prometheus server
  # Key/Value where Key is the sidecar `- name: <Key>`
  # Example:
  #   sidecarContainers:
  #      webserver:
  #        image: nginx
  sidecarContainers: {}

  # sidecarTemplateValues - context to be used in template for sidecarContainers
  # Example:
  #   sidecarTemplateValues: *your-custom-globals
  #   sidecarContainers:
  #     webserver: |-
  #       {{ include "webserver-container-template" . }}
  # Template for `webserver-container-template` might looks like this:
  #   image: "{{ .Values.server.sidecarTemplateValues.repository }}:{{ .Values.server.sidecarTemplateValues.tag }}"
  #   ...
  #
  sidecarTemplateValues: {}

  ## Prometheus server container image
  ##
  image:
    repository: quay.io/prometheus/prometheus
    # if not set appVersion field from Chart.yaml is used
    tag: ""
    # When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value).
    digest: ""
    pullPolicy: IfNotPresent

  ## prometheus server priorityClassName
  ##
  priorityClassName: ""

  ## EnableServiceLinks indicates whether information about services should be injected
  ## into pod's environment variables, matching the syntax of Docker links.
  ## WARNING: the field is unsupported and will be skipped in K8s prior to v1.13.0.
  ##
  enableServiceLinks: true

  ## The URL prefix at which the container can be accessed. Useful in the case the '-web.external-url' includes a slug
  ## so that the various internal URLs are still able to access as they are in the default case.
  ## (Optional)
  prefixURL: ""

  ## External URL which can access prometheus
  ## Maybe same with Ingress host name
  baseURL: ""

  ## Additional server container environment variables
  ##
  ## You specify this manually like you would a raw deployment manifest.
  ## This means you can bind in environment variables from secrets.
  ##
  ## e.g. static environment variable:
  ##  - name: DEMO_GREETING
  ##    value: "Hello from the environment"
  ##
  ## e.g. secret environment variable:
  ## - name: USERNAME
  ##   valueFrom:
  ##     secretKeyRef:
  ##       name: mysecret
  ##       key: username
  env: []

  # List of flags to override default parameters, e.g:
  # - --enable-feature=agent
  # - --storage.agent.retention.max-time=30m
  defaultFlagsOverride: []

  extraFlags:
    - web.enable-lifecycle
    ## web.enable-admin-api flag controls access to the administrative HTTP API which includes functionality such as
    ## deleting time series. This is disabled by default.
    # - web.enable-admin-api
    ##
    ## storage.tsdb.no-lockfile flag controls BD locking
    # - storage.tsdb.no-lockfile
    ##
    ## storage.tsdb.wal-compression flag enables compression of the write-ahead log (WAL)
    # - storage.tsdb.wal-compression

  ## Path to a configuration file on prometheus server container FS
  configPath: /etc/config/prometheus.yml

  ### The data directory used by prometheus to set --storage.tsdb.path
  ### When empty server.persistentVolume.mountPath is used instead
  storagePath: ""

  global:
    ## How frequently to scrape targets by default
    ##
    scrape_interval: 1m
    ## How long until a scrape request times out
    ##
    scrape_timeout: 10s
    ## How frequently to evaluate rules
    ##
    evaluation_interval: 1m
  ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
  ##
  remoteWrite: []
  ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read
  ##
  remoteRead: []

  ## Custom HTTP headers for Liveness/Readiness/Startup Probe
  ##
  ## Useful for providing HTTP Basic Auth to healthchecks
  probeHeaders: []
    # - name: "Authorization"
    #   value: "Bearer ABCDEabcde12345"

  ## Additional Prometheus server container arguments
  ##
  extraArgs: {}

  ## Additional InitContainers to initialize the pod
  ##
  extraInitContainers: []

  ## Additional Prometheus server Volume mounts
  ##
  extraVolumeMounts: []

  ## Additional Prometheus server Volumes
  ##
  extraVolumes: []

  ## Additional Prometheus server hostPath mounts
  ##
  extraHostPathMounts: []
    # - name: certs-dir
    #   mountPath: /etc/kubernetes/certs
    #   subPath: ""
    #   hostPath: /etc/kubernetes/certs
    #   readOnly: true

  extraConfigmapMounts: []
    # - name: certs-configmap
    #   mountPath: /prometheus
    #   subPath: ""
    #   configMap: certs-configmap
    #   readOnly: true

  ## Additional Prometheus server Secret mounts
  # Defines additional mounts with secrets. Secrets must be manually created in the namespace.
  extraSecretMounts: []
    # - name: secret-files
    #   mountPath: /etc/secrets
    #   subPath: ""
    #   secretName: prom-secret-files
    #   readOnly: true

  ## ConfigMap override where fullname is {{.Release.Name}}-{{.Values.server.configMapOverrideName}}
  ## Defining configMapOverrideName will cause templates/server-configmap.yaml
  ## to NOT generate a ConfigMap resource
  ##
  configMapOverrideName: ""

  ## Extra labels for Prometheus server ConfigMap (ConfigMap that holds serverFiles)
  extraConfigmapLabels: {}

  ingress:
    ## If true, Prometheus server Ingress will be created
    ##
    enabled: false

    # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
    # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
    # ingressClassName: nginx

    ## Prometheus server Ingress annotations
    ##
    annotations: {}
    #   kubernetes.io/ingress.class: nginx
    #   kubernetes.io/tls-acme: 'true'

    ## Prometheus server Ingress additional labels
    ##
    extraLabels: {}

    ## Prometheus server Ingress hostnames with optional path
    ## Must be provided if Ingress is enabled
    ##
    hosts: []
    #   - prometheus.domain.com
    #   - domain.com/prometheus

    path: /

    # pathType is only for k8s >= 1.18
    pathType: Prefix

    ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
    extraPaths: []
    # - path: /*
    #   backend:
    #     serviceName: ssl-redirect
    #     servicePort: use-annotation

    ## Prometheus server Ingress TLS configuration
    ## Secrets must be manually created in the namespace
    ##
    tls: []
    #   - secretName: prometheus-server-tls
    #     hosts:
    #       - prometheus.domain.com

  ## Server Deployment Strategy type
  strategy:
    type: Recreate

  ## hostAliases allows adding entries to /etc/hosts inside the containers
  hostAliases: []
  #   - ip: "127.0.0.1"
  #     hostnames:
  #       - "example.com"

  ## Node tolerations for server scheduling to nodes with taints
  ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  ##
  tolerations: []
    # - key: "key"
    #   operator: "Equal|Exists"
    #   value: "value"
    #   effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"

  ## Node labels for Prometheus server pod assignment
  ## Ref: https://kubernetes.io/docs/user-guide/node-selection/
  ##
  nodeSelector: {}

  ## Pod affinity
  ##
  # affinity: {}
  # affinity:
    # nodeAffinity:
    #   requiredDuringSchedulingIgnoredDuringExecution:
    #     nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: eks.amazonaws.com/compute-type
    #           operator: NotIn
    #           values:
    #             - fargate
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/os
              operator: In
              values:
              - linux
            - key: kubernetes.io/arch
              operator: In
              values:
              - amd64
              - arm64
            - key: eks.amazonaws.com/compute-type
              operator: NotIn
              values:
              - fargate
  ## PodDisruptionBudget settings
  ## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
  ##
  podDisruptionBudget:
    enabled: false
    maxUnavailable: 1

  ## Use an alternate scheduler, e.g. "stork".
  ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
  ##
  # schedulerName:

  persistentVolume:
    ## If true, Prometheus server will create/use a Persistent Volume Claim
    ## If false, use emptyDir
    ##
    enabled: true

    ## Prometheus server data Persistent Volume access modes
    ## Must match those of existing PV or dynamic provisioner
    ## Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
    ##
    accessModes:
      - ReadWriteOnce

    ## Prometheus server data Persistent Volume labels
    ##
    labels: {}

    ## Prometheus server data Persistent Volume annotations
    ##
    annotations: {}

    ## Prometheus server data Persistent Volume existing claim name
    ## Requires server.persistentVolume.enabled: true
    ## If defined, PVC must be created manually before volume will be bound
    existingClaim: ""

    ## Prometheus server data Persistent Volume mount root path
    ##
    mountPath: /data

    ## Prometheus server data Persistent Volume size
    ##
    size: 8Gi

    ## Prometheus server data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"

    ## Prometheus server data Persistent Volume Binding Mode
    ## If defined, volumeBindingMode: <volumeBindingMode>
    ## If undefined (the default) or set to null, no volumeBindingMode spec is
    ##   set, choosing the default mode.
    ##
    # volumeBindingMode: ""

    ## Subdirectory of Prometheus server data Persistent Volume to mount
    ## Useful if the volume's root directory is not empty
    ##
    subPath: ""

    ## Persistent Volume Claim Selector
    ## Useful if Persistent Volumes have been provisioned in advance
    ## Ref: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector
    ##
    # selector:
    #  matchLabels:
    #    release: "stable"
    #  matchExpressions:
    #    - { key: environment, operator: In, values: [ dev ] }

    ## Persistent Volume Name
    ## Useful if Persistent Volumes have been provisioned in advance and you want to use a specific one
    ##
    # volumeName: ""

  emptyDir:
    ## Prometheus server emptyDir volume size limit
    ##
    sizeLimit: ""

  ## Annotations to be added to Prometheus server pods
  ##
  podAnnotations: {}
    # iam.amazonaws.com/role: prometheus

  ## Labels to be added to Prometheus server pods
  ##
  podLabels: {}

  ## Prometheus AlertManager configuration
  ##
  alertmanagers: []

  ## Specify if a Pod Security Policy for node-exporter must be created
  ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  ##
  podSecurityPolicy:
    annotations: {}
      ## Specify pod annotations
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl
      ##
      # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
      # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
      # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

  ## Use a StatefulSet if replicaCount needs to be greater than 1 (see below)
  ##
  replicaCount: 1

  ## Annotations to be added to deployment
  ##
  deploymentAnnotations: {}

  statefulSet:
    ## If true, use a statefulset instead of a deployment for pod management.
    ## This allows to scale replicas to more than 1 pod
    ##
    enabled: false

    annotations: {}
    labels: {}
    podManagementPolicy: OrderedReady

    ## Alertmanager headless service to use for the statefulset
    ##
    headless:
      annotations: {}
      labels: {}
      servicePort: 80
      ## Enable gRPC port on service to allow auto discovery with thanos-querier
      gRPC:
        enabled: false
        servicePort: 10901
        # nodePort: 10901

  ## Prometheus server readiness and liveness probe initial delay and timeout
  ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
  ##
  tcpSocketProbeEnabled: false
  probeScheme: HTTP
  readinessProbeInitialDelay: 30
  readinessProbePeriodSeconds: 5
  readinessProbeTimeout: 4
  readinessProbeFailureThreshold: 3
  readinessProbeSuccessThreshold: 1
  livenessProbeInitialDelay: 30
  livenessProbePeriodSeconds: 15
  livenessProbeTimeout: 10
  livenessProbeFailureThreshold: 3
  livenessProbeSuccessThreshold: 1
  startupProbe:
    enabled: false
    periodSeconds: 5
    failureThreshold: 30
    timeoutSeconds: 10

  ## Prometheus server resource requests and limits
  ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources: {}
    # limits:
    #   cpu: 500m
    #   memory: 512Mi
    # requests:
    #   cpu: 500m
    #   memory: 512Mi

  # Required for use in managed kubernetes clusters (such as AWS EKS) with custom CNI (such as calico),
  # because control-plane managed by AWS cannot communicate with pods' IP CIDR and admission webhooks are not working
  ##
  hostNetwork: false

  # When hostNetwork is enabled, this will set to ClusterFirstWithHostNet automatically
  dnsPolicy: ClusterFirst

  # Use hostPort
  # hostPort: 9090

  ## Vertical Pod Autoscaler config
  ## Ref: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler
  verticalAutoscaler:
    ## If true a VPA object will be created for the controller (either StatefulSet or Deployemnt, based on above configs)
    enabled: false
    # updateMode: "Auto"
    # containerPolicies:
    # - containerName: 'prometheus-server'

  # Custom DNS configuration to be added to prometheus server pods
  dnsConfig: {}
    # nameservers:
    #   - 1.2.3.4
    # searches:
    #   - ns1.svc.cluster-domain.example
    #   - my.dns.search.suffix
    # options:
    #   - name: ndots
    #     value: "2"
  #   - name: edns0

  ## Security context to be added to server pods
  ##
  securityContext:
    runAsUser: 65534
    runAsNonRoot: true
    runAsGroup: 65534
    fsGroup: 65534

  ## Security context to be added to server container
  ##
  containerSecurityContext: {}

  service:
    ## If false, no Service will be created for the Prometheus server
    ##
    enabled: true

    annotations: {}
    labels: {}
    clusterIP: ""

    ## List of IP addresses at which the Prometheus server service is available
    ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
    ##
    externalIPs: []

    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    servicePort: 80
    sessionAffinity: None
    type: ClusterIP

    ## Enable gRPC port on service to allow auto discovery with thanos-querier
    gRPC:
      enabled: false
      servicePort: 10901
      # nodePort: 10901

    ## If using a statefulSet (statefulSet.enabled=true), configure the
    ## service to connect to a specific replica to have a consistent view
    ## of the data.
    statefulsetReplica:
      enabled: false
      replica: 0

  ## Prometheus server pod termination grace period
  ##
  terminationGracePeriodSeconds: 300

  ## Prometheus data retention period (default if not specified is 15 days)
  ##
  retention: "15d"

## Prometheus server ConfigMap entries for rule files (allow prometheus labels interpolation)
ruleFiles: {}

## Prometheus server ConfigMap entries
##
serverFiles:
  ## Alerts configuration
  ## Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
  alerting_rules.yml: {}
  # groups:
  #   - name: Instances
  #     rules:
  #       - alert: InstanceDown
  #         expr: up == 0
  #         for: 5m
  #         labels:
  #           severity: page
  #         annotations:
  #           description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
  #           summary: 'Instance {{ $labels.instance }} down'
  ## DEPRECATED DEFAULT VALUE, unless explicitly naming your files, please use alerting_rules.yml
  alerts: {}

  ## Records configuration
  ## Ref: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/
  recording_rules.yml: {}
  ## DEPRECATED DEFAULT VALUE, unless explicitly naming your files, please use recording_rules.yml
  rules: {}

  prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml
      - /etc/config/alerting_rules.yml
    ## Below two files are DEPRECATED will be removed from this default values file
      - /etc/config/rules
      - /etc/config/alerts

    scrape_configs:
      - job_name: prometheus
        static_configs:
          - targets:
            - localhost:9090

      # A scrape configuration for running Prometheus on a Kubernetes cluster.
      # This uses separate scrape configs for cluster components (i.e. API server, node)
      # and services to allow each to use different authentication configs.
      #
      # Kubernetes labels will be added as Prometheus labels on metrics via the
      # `labelmap` relabeling action.

      # Scrape config for API servers.
      #
      # Kubernetes exposes API servers as endpoints to the default/kubernetes
      # service so this uses `endpoints` role and uses relabelling to only keep
      # the endpoints associated with the default/kubernetes service using the
      # default named port `https`. This works for single API server deployments as
      # well as HA API server deployments.
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
          - role: endpoints

        # Default to scraping over https. If required, just disable this or change to
        # `http`.
        scheme: https

        # This TLS & bearer token file config is used to connect to the actual scrape
        # endpoints for cluster components. This is separate to discovery auth
        # configuration because discovery & scraping are two separate concerns in
        # Prometheus. The discovery auth config is automatic if Prometheus runs inside
        # the cluster. Otherwise, more config options have to be provided within the
        # <kubernetes_sd_config>.
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          # If your node certificates are self-signed or use a different CA to the
          # master CA, then disable certificate verification below. Note that
          # certificate verification is an integral part of a secure infrastructure
          # so this should only be disabled in a controlled environment. You can
          # disable certificate verification by uncommenting the line below.
          #
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        # Keep only the default/kubernetes service endpoints for the https port. This
        # will add targets for each API server which Kubernetes adds an endpoint to
        # the default/kubernetes service.
        relabel_configs:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        # Default to scraping over https. If required, just disable this or change to
        # `http`.
        scheme: https

        # This TLS & bearer token file config is used to connect to the actual scrape
        # endpoints for cluster components. This is separate to discovery auth
        # configuration because discovery & scraping are two separate concerns in
        # Prometheus. The discovery auth config is automatic if Prometheus runs inside
        # the cluster. Otherwise, more config options have to be provided within the
        # <kubernetes_sd_config>.
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          # If your node certificates are self-signed or use a different CA to the
          # master CA, then disable certificate verification below. Note that
          # certificate verification is an integral part of a secure infrastructure
          # so this should only be disabled in a controlled environment. You can
          # disable certificate verification by uncommenting the line below.
          #
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
          - role: node

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics


      - job_name: 'kubernetes-nodes-cadvisor'

        # Default to scraping over https. If required, just disable this or change to
        # `http`.
        scheme: https

        # This TLS & bearer token file config is used to connect to the actual scrape
        # endpoints for cluster components. This is separate to discovery auth
        # configuration because discovery & scraping are two separate concerns in
        # Prometheus. The discovery auth config is automatic if Prometheus runs inside
        # the cluster. Otherwise, more config options have to be provided within the
        # <kubernetes_sd_config>.
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          # If your node certificates are self-signed or use a different CA to the
          # master CA, then disable certificate verification below. Note that
          # certificate verification is an integral part of a secure infrastructure
          # so this should only be disabled in a controlled environment. You can
          # disable certificate verification by uncommenting the line below.
          #
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
          - role: node

        # This configuration will work only on kubelet 1.7.3+
        # As the scrape endpoints for cAdvisor have changed
        # if you are using older version you need to change the replacement to
        # replacement: /api/v1/nodes/$1:4194/proxy/metrics
        # more info here https://github.com/coreos/prometheus-operator/issues/633
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

        # Metric relabel configs to apply to samples before ingestion.
        # [Metric Relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs)
        # metric_relabel_configs:
        # - action: labeldrop
        #   regex: (kubernetes_io_hostname|failure_domain_beta_kubernetes_io_region|beta_kubernetes_io_os|beta_kubernetes_io_arch|beta_kubernetes_io_instance_type|failure_domain_beta_kubernetes_io_zone)

      # Scrape config for service endpoints.
      #
      # The relabeling allows the actual service scrape endpoint to be configured
      # via the following annotations:
      #
      # * `prometheus.io/scrape`: Only scrape services that have a value of
      # `true`, except if `prometheus.io/scrape-slow` is set to `true` as well.
      # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
      # to set this to `https` & most likely set the `tls_config` of the scrape config.
      # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
      # * `prometheus.io/port`: If the metrics are exposed on a different port to the
      # service then set this appropriately.
      # * `prometheus.io/param_<parameter>`: If the metrics endpoint uses parameters
      # then you can set any parameter
      - job_name: 'kubernetes-service-endpoints'
        honor_labels: true

        kubernetes_sd_configs:
          - role: endpoints

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
            action: drop
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+?)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
            replacement: __param_$1
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: service
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: node

      # Scrape config for slow service endpoints; same as above, but with a larger
      # timeout and a larger interval
      #
      # The relabeling allows the actual service scrape endpoint to be configured
      # via the following annotations:
      #
      # * `prometheus.io/scrape-slow`: Only scrape services that have a value of `true`
      # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
      # to set this to `https` & most likely set the `tls_config` of the scrape config.
      # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
      # * `prometheus.io/port`: If the metrics are exposed on a different port to the
      # service then set this appropriately.
      # * `prometheus.io/param_<parameter>`: If the metrics endpoint uses parameters
      # then you can set any parameter
      - job_name: 'kubernetes-service-endpoints-slow'
        honor_labels: true

        scrape_interval: 5m
        scrape_timeout: 30s

        kubernetes_sd_configs:
          - role: endpoints

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+?)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
            replacement: __param_$1
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: service
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: node

      - job_name: 'prometheus-pushgateway'
        honor_labels: true

        kubernetes_sd_configs:
          - role: service

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: pushgateway

      # Example scrape config for probing services via the Blackbox Exporter.
      #
      # The relabeling allows the actual service scrape endpoint to be configured
      # via the following annotations:
      #
      # * `prometheus.io/probe`: Only probe services that have a value of `true`
      - job_name: 'kubernetes-services'
        honor_labels: true

        metrics_path: /probe
        params:
          module: [http_2xx]

        kubernetes_sd_configs:
          - role: service

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service

      # Example scrape config for pods
      #
      # The relabeling allows the actual pod scrape endpoint to be configured via the
      # following annotations:
      #
      # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`,
      # except if `prometheus.io/scrape-slow` is set to `true` as well.
      # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
      # to set this to `https` & most likely set the `tls_config` of the scrape config.
      # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
      # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
      - job_name: 'kubernetes-pods'
        honor_labels: true

        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
            action: drop
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
            action: replace
            regex: (https?)
            target_label: __scheme__
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
            action: replace
            regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
            replacement: '[$2]:$1'
            target_label: __address__
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
            action: replace
            regex: (\d+);((([0-9]+?)(\.|$)){4})
            replacement: $2:$1
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
            replacement: __param_$1
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_phase]
            regex: Pending|Succeeded|Failed|Completed
            action: drop

      # Example Scrape config for pods which should be scraped slower. An useful example
      # would be stackriver-exporter which queries an API on every scrape of the pod
      #
      # The relabeling allows the actual pod scrape endpoint to be configured via the
      # following annotations:
      #
      # * `prometheus.io/scrape-slow`: Only scrape pods that have a value of `true`
      # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
      # to set this to `https` & most likely set the `tls_config` of the scrape config.
      # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
      # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
      - job_name: 'kubernetes-pods-slow'
        honor_labels: true

        scrape_interval: 5m
        scrape_timeout: 30s

        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
            action: replace
            regex: (https?)
            target_label: __scheme__
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
            action: replace
            regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
            replacement: '[$2]:$1'
            target_label: __address__
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
            action: replace
            regex: (\d+);((([0-9]+?)(\.|$)){4})
            replacement: $2:$1
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
            replacement: __param_$1
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_phase]
            regex: Pending|Succeeded|Failed|Completed
            action: drop

# adds additional scrape configs to prometheus.yml
# must be a string so you have to add a | after extraScrapeConfigs:
# example adds prometheus-blackbox-exporter scrape config
extraScrapeConfigs: ""
  # - job_name: 'prometheus-blackbox-exporter'
  #   metrics_path: /probe
  #   params:
  #     module: [http_2xx]
  #   static_configs:
  #     - targets:
  #       - https://example.com
  #   relabel_configs:
  #     - source_labels: [__address__]
  #       target_label: __param_target
  #     - source_labels: [__param_target]
  #       target_label: instance
  #     - target_label: __address__
  #       replacement: prometheus-blackbox-exporter:9115

# Adds option to add alert_relabel_configs to avoid duplicate alerts in alertmanager
# useful in H/A prometheus with different external labels but the same alerts
alertRelabelConfigs: {}
  # alert_relabel_configs:
  # - source_labels: [dc]
  #   regex: (.+)\d+
  #   target_label: dc

networkPolicy:
  ## Enable creation of NetworkPolicy resources.
  ##
  enabled: false

# Force namespace of namespaced resources
forceNamespace: ""

# Extra manifests to deploy as an array
extraManifests: []
  # - apiVersion: v1
  #   kind: ConfigMap
  #   metadata:
  #   labels:
  #     name: prometheus-extra
  #   data:
  #     extra-data: "value"

# Configuration of subcharts defined in Chart.yaml

## alertmanager sub-chart configurable values
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/alertmanager
##
alertmanager:
  ## If false, alertmanager will not be installed
  ##
  enabled: true

  persistence:
    size: 2Gi

  podSecurityContext:
    runAsUser: 65534
    runAsNonRoot: true
    runAsGroup: 65534
    fsGroup: 65534

## kube-state-metrics sub-chart configurable values
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics
##
kube-state-metrics:
  ## If false, kube-state-metrics sub-chart will not be installed
  ##
  enabled: true

## promtheus-node-exporter sub-chart configurable values
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter
##
prometheus-node-exporter:
  ## If false, node-exporter will not be installed
  ##
  enabled: true

  rbac:
    pspEnabled: false

  containerSecurityContext:
    allowPrivilegeEscalation: false
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/os
              operator: In
              values:
              - linux
            - key: kubernetes.io/arch
              operator: In
              values:
              - amd64
              - arm64
            - key: eks.amazonaws.com/compute-type
              operator: NotIn
              values:
              - fargate

## pprometheus-pushgateway sub-chart configurable values
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway
##
prometheus-pushgateway:
  ## If false, pushgateway will not be installed
  ##
  enabled: true

  # Optional service annotations
  serviceAnnotations:
    prometheus.io/probe: pushgateway

Note: you have to add nodeAffinity for node exporter



affinity:

    nodeAffinity:

      requiredDuringSchedulingIgnoredDuringExecution:

        nodeSelectorTerms:

          - matchExpressions:

            - key: kubernetes.io/os

              operator: In

              values:

              - linux

            - key: kubernetes.io/arch

              operator: In

              values:

              - amd64

              - arm64

            - key: eks.amazonaws.com/compute-type

              operator: NotIn

              values:

              - fargate

Ingress for prometheus url

Lets add the ingress for prometheus in the following way-



apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  annotations:

    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":

      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:2343668766434:certificate/c25d25f3-78ae-4197-a806-1882f6b947dc

    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'

    alb.ingress.kubernetes.io/scheme: internet-facing

    alb.ingress.kubernetes.io/success-codes: 200,404,301,302

    alb.ingress.kubernetes.io/target-type: ip

  finalizers:


ingress.k8s.aws/resources
name: prometheus-server
namespace: prometheus
spec:
ingressClassName: alb
rules:
http:
  paths:


backend:
  service:
    name: prometheus-server
    port:
      number: 80
path: /
pathType: Prefix
backend:
  service:
    name: prometheus-server
    port:
      number: 80
path: /
pathType: Prefix

AWS Managed Grafana:

Add this prometheus url as a datasource on grafana.

That's it! . It's ready now to create dashboard.

AWS EKS Setup with eksctl & Argo CD installation, configuration & deploy app with ArgoCD & Kustomize

M.M.Monirul Islam — Mon, 23 May 2022 15:47:50 +0000

Setting up a production Kubernetes service on AWS EKS

The easiest way to create an EKS on AWS is to use eksctl. And it is recommended to create a bastion server in AWS and run it there rather than a laptop as the environment to run the eksctl cli.

Create AWS User

First, create an admin user so that it can be used programatically in AWS IAM, and obtain the AWS Access Key ID and AWS Secret Access Key of the user.

Create Bastion Server

Let's create a bastion server in AWS. Even if the instance type is t3.small, it is sufficient.

Install kubectl

Install kubectl of desired kubernetes version



$ curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
$ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
$ kubectl version --short --client

Reference link: https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html

Install aws cli

To use eksctl, We need to set the credential of the AWS user.



$ apt install unzip
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ ./aws/install

Reference link to install the aws cli:
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

AWS cli configuration settings

Now set configuration in aws cli in the following way-



$ aws configure
AWS Access Key ID [None]: ~~~
AWS Secret Access Key [None]: ~~~
Default region name [None]: ap-southeast-1
Default output format [None]: json

Reference link: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html

Install eksctl cli



$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
$ mv /tmp/eksctl /usr/local/bin
$ eksctl version
0.98.0

Reference link: https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html

Create EKS Cluster

Create EKS Cluster with eksctl cli.



$ eksctl create cluster \
--version 1.21 \
--name eks-monirul-cluster \
--vpc-nat-mode HighlyAvailable \
--node-private-networking \
--region ap-southeast-1 \
--node-type t3.medium \
--nodes 2 \
--with-oidc \
--ssh-access \
--ssh-public-key monirul \
--managed

Here,
version: the Kubernetes version to use

vpc-nat-mode: All outbounds of kubernetes go out through the nat gateway. The default option is single, so only one is created. In a development environment, it may not matter, but in production, we must use the HighAvailable option to create one for each subnet.

node-private-networking: If this option is not present, a node group is created in the public subnet. Use this option so that it is created in a private subnet for security.

node-type: the instance type of the node to be created

nodes: the number of nodes to be created

Check if the cluster has been successfully created



$ kubectl get nodes -o wide

EKS security related settings

Up to this point, a Kubernets cluster is created in a secure way considering HA for commercial use. But for security, we need to do one more thing. The current state is that the cluster endpoint is allowed to be public.
https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes/
https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html

To restrict this, allow private access and modify kubernetes to issue commands only to the bastion server by limiting the CIDR block. Enter the public IPv4 address of the bastion server.



$ eksctl utils update-cluster-endpoints --cluster=eks-monirul-cluster --private-access=true --public-access=true --approve
$ eksctl utils set-public-access-cidrs --cluster=eks-monirul-cluster 1.1.1.1/32 --approve

1.1.1.1/32 is the address of the bastion server.

In fact, EKS Cluster creation can be created at once by creating a yaml file with the --dry-run option of eksctl and giving all options from creation to EKS security-related settings.

Now, if we install Istio in the EKS Cluster created in this way and deploy the service, we can use it immediately.

Argo CD installation, setup & deploy a simple app with kustomize

ArgoCD monitors changes in Kubernetes manifests managed by GitOps, and plays a role in maintaining the form deployed in the actual cluster in the same way.

Argo CD installation

https://argo-cd.readthedocs.io/en/stable/getting_started/
Install Argo CD on kubernetes cluster

In production, install the HA version of the Argo CD.



$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

Install Argo CD CLI



$ curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
$ chmod +x /usr/local/bin/argocd

Argo CD service exposure

Argo CD does not expose the server to the outside by default. Change the service type to LoadBalancer as shown below and expose it to the outside.



$ kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Change admin password

Argo CD stores the initial password of the initial admin account as the secret of kubernetes. Get the password as below.



$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
jaIyQ3MMuLnl6h0l



$ kubectl get svc argocd-server -n argocd

And log in. Username is admin



$ argocd login <ARGOCD_SERVER_DOMAIN/URL>

Update the password of the admin user after first login.



$ argocd account update-password

Argo CD polls the Git repository once every 3 minutes to check the difference from the actual kubernetes cluster. Therefore, if we are unlucky during distribution, we have to wait up to 3 minutes before Argo CD distributes the changed image. If we want to eliminate the delay caused by polling like this, we can create a webhook with Argo CD in the Git repository. Here is the link:
https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/

And, usually Argo CD has only a specific IP inbound in the security group to prevent access except for internal developers or operators. At this time, if we have created a webhook as above, we must also open Github's webhook-related API inbound to the Argo CD's load balancer.

The link below contains information about the IP address of GitHub,
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses
We can check the IP that needs to be put in the actual inbound below. Just put the list in hooks.
https://api.github.com/meta

If we configure to send notification to Slack using Argo CD Notification, the development team can receive a notification when the deployment is not successful.

There is also a way to set up the project by using the ApplicationSet of the Argo CD. If we used App of Apps in the past, a more advanced concept here is Application Sets.

https://opengitops.dev/
https://github.com/open-gitops/documents

Currently, kustomize is used for configuration management of Kubernetes, and kustomize is deployed from Argo CD. If we are using branching in kustomize or helm, it might be helpful to read Stop Using Branches for Deploying to Different GitOps Environments.

Kustomize yaml-

kustomization.yaml



apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: kustomize-monirul-

resources:
- nginx-deployment.yaml
- nginx-svc.yaml

Deployment definition file: nginx-deployment.yaml



apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx

Service definition file: nginx-svc.yaml



apiVersion: v1

kind: Service

metadata:

  labels:

    app: nginx

  name: nginx

spec:

  ports:


port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP

ArgoCD configuration to deploy simple app:

Screenshot of Argo CD:

Provisioning Amazon Elastic Kubernetes Service (Amazon EKS) with Terraform

M.M.Monirul Islam — Wed, 18 May 2022 15:32:59 +0000

AWS EKS is a managed container service to run and scale Kubernetes applications in the cloud or on-premises.

HashiCorp Terraform is an Infrastructure as Code (IaC) tool that lets us define both cloud and on-prem resources in human-readable configuration files that can version, reuse, and share.

Amazon EKS Cluster using Terraform:

This Repository will be used to keep infrastructure configuration for eks cluster

Prerequisite

AWS Account
Terraform
AWS CLI

To configure AWS CLI, We need to enter AWS Access Key ID, Secret Access Key, region and output format. Please note proper privillage is required to create eks cluster resources.

$ aws configure
AWS Access Key ID [None]: AWS_ACCESS_KEY_ID
AWS Secret Access Key [None]: AWS_SECRET_ACCESS_KEY
Default region name [None]: AWS_REGION
Default output format [None]: json

Terraform Initial Setup Configuration

Need to create an AWS provider. It allows to interact with the AWS resources, such as VPC, EKS, S3, EC2, and many others.

providers.tf

terraform {
  required_version = ">= 1.1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region = var.region

  access_key = var.aws_access_key
  secret_key = var.aws_secret_key

  # other options for authentication
}

Terraform State Setup

Now, We need to create terraform backend to specify the location of the backend Terraform state file on S3.
Remote state is storing that state file remotely, rather than on my local filesystem.
backend.tf

terraform {
  backend "s3" {
    bucket  = "mondev-terraform-states"
    key     = "terraform-aws-eks-mondev.tfstate"
    region  = "ap-southeast-1"
    encrypt = true
  }
}

Network Infrastructure Setup

Setting up the VPC, Subnets, Security Groups, etc.
Amazon EKS requires subnets must be in at least two different availability zones.

AWS VPC (Virtual Private Cloud).
Two public and two private Subnets in different availability zones.
Internet Gateway to provide internet access for services within VPC.
NAT Gateway in public subnets. It is used in private subnets to allow services to connect to the internet.
Routing Tables and associate subnets with them. Add required routing rules.
Security Groups and associate subnets with them. Add required routing rules.

vpc.tf

# VPC
resource "aws_vpc" "mondev" {
  cidr_block = var.vpc_cidr

  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name                                           = "${var.project}-vpc",
    "kubernetes.io/cluster/${var.project}-cluster" = "shared"
  }
}

# Public Subnets
resource "aws_subnet" "public" {
  count = var.availability_zones_count

  vpc_id            = aws_vpc.mondev.id
  cidr_block        = cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name                                           = "${var.project}-public-sg"
    "kubernetes.io/cluster/${var.project}-cluster" = "shared"
    "kubernetes.io/role/elb"                       = 1
  }

  map_public_ip_on_launch = true
}

# Private Subnets
resource "aws_subnet" "private" {
  count = var.availability_zones_count

  vpc_id            = aws_vpc.mondev.id
  cidr_block        = cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, count.index + var.availability_zones_count)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name                                           = "${var.project}-private-sg"
    "kubernetes.io/cluster/${var.project}-cluster" = "shared"
    "kubernetes.io/role/internal-elb"              = 1
  }
}

# Internet Gateway
resource "aws_internet_gateway" "mondev" {
  vpc_id = aws_vpc.mondev.id

  tags = {
    "Name" = "${var.project}-igw"
  }

  depends_on = [aws_vpc.mondev]
}

# Route Table(s)
# Route the public subnet traffic through the IGW
resource "aws_route_table" "main" {
  vpc_id = aws_vpc.mondev.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.mondev.id
  }

  tags = {
    Name = "${var.project}-Default-rt"
  }
}

# Route table and subnet associations
resource "aws_route_table_association" "internet_access" {
  count = var.availability_zones_count

  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.main.id
}

# NAT Elastic IP
resource "aws_eip" "main" {
  vpc = true

  tags = {
    Name = "${var.project}-ngw-ip"
  }
}

# NAT Gateway
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.main.id
  subnet_id     = aws_subnet.public[0].id

  tags = {
    Name = "${var.project}-ngw"
  }
}

# Add route to route table
resource "aws_route" "main" {
  route_table_id         = aws_vpc.mondev.default_route_table_id
  nat_gateway_id         = aws_nat_gateway.main.id
  destination_cidr_block = "0.0.0.0/0"
}

# Security group for public subnet
resource "aws_security_group" "public_sg" {
  name   = "${var.project}-Public-sg"
  vpc_id = aws_vpc.mondev.id

  tags = {
    Name = "${var.project}-Public-sg"
  }
}

# Security group traffic rules
resource "aws_security_group_rule" "sg_ingress_public_443" {
  security_group_id = aws_security_group.public_sg.id
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "sg_ingress_public_80" {
  security_group_id = aws_security_group.public_sg.id
  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "sg_egress_public" {
  security_group_id = aws_security_group.public_sg.id
  type              = "egress"
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  cidr_blocks       = ["0.0.0.0/0"]
}

# Security group for data plane
resource "aws_security_group" "data_plane_sg" {
  name   = "${var.project}-Worker-sg"
  vpc_id = aws_vpc.mondev.id

  tags = {
    Name = "${var.project}-Worker-sg"
  }
}

# Security group traffic rules
resource "aws_security_group_rule" "nodes" {
  description       = "Allow nodes to communicate with each other"
  security_group_id = aws_security_group.data_plane_sg.id
  type              = "ingress"
  from_port         = 0
  to_port           = 65535
  protocol          = "-1"
  cidr_blocks       = flatten([cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 0), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 1), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 2), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 3)])
}

resource "aws_security_group_rule" "nodes_inbound" {
  description       = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
  security_group_id = aws_security_group.data_plane_sg.id
  type              = "ingress"
  from_port         = 1025
  to_port           = 65535
  protocol          = "tcp"
  cidr_blocks       = flatten([cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 2), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 3)])
  # cidr_blocks       = flatten([var.private_subnet_cidr_blocks])
}

resource "aws_security_group_rule" "node_outbound" {
  security_group_id = aws_security_group.data_plane_sg.id
  type              = "egress"
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  cidr_blocks       = ["0.0.0.0/0"]
}

# Security group for control plane
resource "aws_security_group" "control_plane_sg" {
  name   = "${var.project}-ControlPlane-sg"
  vpc_id = aws_vpc.mondev.id

  tags = {
    Name = "${var.project}-ControlPlane-sg"
  }
}

# Security group traffic rules
resource "aws_security_group_rule" "control_plane_inbound" {
  security_group_id = aws_security_group.control_plane_sg.id
  type              = "ingress"
  from_port         = 0
  to_port           = 65535
  protocol          = "tcp"
  cidr_blocks       = flatten([cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 0), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 1), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 2), cidrsubnet(var.vpc_cidr, var.subnet_cidr_bits, 3)])
  # cidr_blocks       = flatten([var.private_subnet_cidr_blocks, var.public_subnet_cidr_blocks])
}

resource "aws_security_group_rule" "control_plane_outbound" {
  security_group_id = aws_security_group.control_plane_sg.id
  type              = "egress"
  from_port         = 0
  to_port           = 65535
  protocol          = "-1"
  cidr_blocks       = ["0.0.0.0/0"]
}

EKS Cluster Setup

Creating EKS cluster. Kubernetes clusters managed by Amazon EKS make calls to other AWS services on your behalf to manage the resources that we use with the service. For example, EKS will create an Auto Scaling Groups for each instance group if we use managed nodes.

Setting up the IAM Roles and Policies for EKS: EKS requires a few IAM Roles with relevant Policies to be pre-defined to operate correctly.

IAM Role: Create Role with the needed permissions that Amazon EKS will use to create AWS resources for Kubernetes clusters and interact with AWS APIs.

IAM Policy: Attach the trusted Policy (AmazonEKSClusterPolicy) which will allow Amazon EKS to assume and use this role.

eks-cluster.tf

# EKS Cluster
resource "aws_eks_cluster" "mondev" {
  name     = "${var.project}-cluster"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.22"

  vpc_config {
    # security_group_ids      = [aws_security_group.eks_cluster.id, aws_security_group.eks_nodes.id] # already applied to subnet
    subnet_ids              = flatten([aws_subnet.public[*].id, aws_subnet.private[*].id])
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = ["0.0.0.0/0"]
  }

  tags = merge(
    var.tags
  )

  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy
  ]
}


# EKS Cluster IAM Role
resource "aws_iam_role" "cluster" {
  name = "${var.project}-Cluster-Role"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster.name
}


# EKS Cluster Security Group
resource "aws_security_group" "eks_cluster" {
  name        = "${var.project}-cluster-sg"
  description = "Cluster communication with worker nodes"
  vpc_id      = aws_vpc.mondev.id

  tags = {
    Name = "${var.project}-cluster-sg"
  }
}

resource "aws_security_group_rule" "cluster_inbound" {
  description              = "Allow worker nodes to communicate with the cluster API Server"
  from_port                = 443
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_cluster.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 443
  type                     = "ingress"
}

resource "aws_security_group_rule" "cluster_outbound" {
  description              = "Allow cluster API Server to communicate with the worker nodes"
  from_port                = 1024
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_cluster.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 65535
  type                     = "egress"

Node Groups (Managed) Setup

Creating a Node Group(s) to run application workload.
IAM Role: Similar to the EKS cluster, before we create worker node group, we must create IAM role with needed permissions for the node group to communicate with other AWS services.

IAM Policy: Attach the trusted Policy (AmazonEKSWorkerNodePolicy) which will allow amazon EC2 to assume and using this role. Also, attach the AWS managed permission Policy (AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly).

node-groups.tf

# EKS Node Groups
resource "aws_eks_node_group" "mondev" {
  cluster_name    = aws_eks_cluster.mondev.name
  node_group_name = var.project
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = aws_subnet.private[*].id

  scaling_config {
    desired_size = 2
    max_size     = 5
    min_size     = 1
  }

  ami_type       = "AL2_x86_64" # AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM
  capacity_type  = "ON_DEMAND"  # ON_DEMAND, SPOT
  disk_size      = 20
  instance_types = ["t2.medium"]

  tags = merge(
    var.tags
  )

  depends_on = [
    aws_iam_role_policy_attachment.node_AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node_AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node_AmazonEC2ContainerRegistryReadOnly,
  ]
}


# EKS Node IAM Role
resource "aws_iam_role" "node" {
  name = "${var.project}-Worker-Role"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.node.name
}


# EKS Node Security Group
resource "aws_security_group" "eks_nodes" {
  name        = "${var.project}-node-sg"
  description = "Security group for all nodes in the cluster"
  vpc_id      = aws_vpc.mondev.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name                                           = "${var.project}-node-sg"
    "kubernetes.io/cluster/${var.project}-cluster" = "owned"
  }
}

resource "aws_security_group_rule" "nodes_internal" {
  description              = "Allow nodes to communicate with each other"
  from_port                = 0
  protocol                 = "-1"
  security_group_id        = aws_security_group.eks_nodes.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 65535
  type                     = "ingress"
}

resource "aws_security_group_rule" "nodes_cluster_inbound" {
  description              = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
  from_port                = 1025
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_nodes.id
  source_security_group_id = aws_security_group.eks_cluster.id
  to_port                  = 65535
  type                     = "ingress"
}

Terraform Variables

Creating IAM user with administrator access to the AWS account, and get access key and secret key for authentication.
variables.tf

variable "aws_access_key" {
  description = "AWS access key"
  type        = string
}

variable "aws_secret_key" {
  description = "AWS secret key"
  type        = string
}

variable "region" {
  description = "The aws region. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html"
  type        = string
  default     = "ap-southeast-1"
}

variable "availability_zones_count" {
  description = "The number of AZs."
  type        = number
  default     = 2
}

variable "project" {
  description = "MonirulProject"
  type        = string
}

variable "vpc_cidr" {
  description = "The CIDR block for the VPC. Default value is a valid CIDR, but not acceptable by AWS and should be overridden"
  type        = string
  default     = "10.0.0.0/16"
}

variable "subnet_cidr_bits" {
  description = "The number of subnet bits for the CIDR. For example, specifying a value 8 for this parameter will create a CIDR with a mask of /24."
  type        = number
  default     = 8
}

variable "tags" {
  description = "A map of tags to add to all resources"
  type        = map(string)
  default = {
    "Project"     = "MonirulProject"
    "Environment" = "Development"
    "Owner"       = "Monirul"
  }
}

Set terraform variables values as per requirements.
terraform.tfvars

aws_access_key = "aaaaaaaaaaaaaa"
aws_secret_key = "bbbbbbbbbbbbbbbbbbbbb"

region                   = "ap-southeast-1"
availability_zones_count = 2

project = "MonirulProject"

vpc_cidr         = "10.0.0.0/16"
subnet_cidr_bits = 8

And, terraform data sources as well.
data-sources.tf

data "aws_availability_zones" "available" {
  state = "available"
}

Launch EKS Infrastructure

Once we have finished declaring the resources, we can deploy all resources.

$ terraform init
$ terraform plan
$ terraform apply

Output

$ terraform output

Project Structure

Cluster
|-- README.md
|-- backend.tf
|-- data-sources.tf
|-- eks-cluster.tf
|-- node-groups.tf
|-- outputs.tf
|-- providers.tf
|-- terraform.tfvars
|-- variables.tf
|-- vpc.tf

Access Cluster and create different namespace if required

aws eks --region ap-southeast-1 update-kubeconfig --name MonirulProject-cluster
kubectl create ns dev && kubectl create ns stg && kubectl create ns prd

Availability zone (Pod Topology)

Clean up workspace

$ terraform destroy

Workspaces for multiple environments

To manage multiple distinct sets of infrastructure resources/environments.
Instead of creating a new directory for each environment to manage we need to just create workspace and use them.

$ terraform workspace new dev
$ terraform workspace new stg
$ terraform workspace new prd
$ terraform workspace list

Auto Scale Kubernetes Pods for Microservices

M.M.Monirul Islam — Mon, 09 May 2022 09:00:50 +0000

In Kubernetes, autoscaling prevents over provisioning resources for microservices running in a cluster. Here is the procedure to set up horizontal and vertical scaling.

In Kubernetes, cluster capacity planning is critical to avoid overprovisioned or under provisioned infrastructure. IT admins need a reliable and cost-effective way to maintain operational clusters and pods in high-load situations and to scale infrastructure automatically to meet resource requirements.

We know that, Kubernetes supports 3 different types of autoscaling:

1. Vertical Pod Autoscaler (VPA). Increases or decreases the resource limits on the pod.
2. Horizontal Pod Autoscaler (HPA). Increases or decreases the number of pod instances.
3. Cluster Autoscaler (CA). Increases or decreases the nodes in the node pool, based on pod
scheduling.

I’m going to focuses on the Horizontal and Vertical options, as I will be working on a pod level, not a node level.

Set up a microservice in a Kubernetes cluster:

To get started, let's create a REST API to deploy as a microservice in containers on Kubernetes. To take this deeper, we can first create the REST API -- written in Go, as presented below -- which deploys a microservice on Kubernetes. Save the below content in a file named deployment.yml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: microsvcmonirul
  namespace: monirul
spec:
selector:
   matchLabels:
     run: microsvcmonirul
replicas: 1
template:
   metadata:
  labels:
    run: microsvcmonirul
spec:
  containers:
  - name: microsvcmonirul
image: "monirul87/microsvcmonirul-1.0.3" ports:
- containerPort: 8080
         resources:
          requests:
            memory: "64Mi"
            cpu: "125m"
          limits:
            memory: "128Mi"
            cpu: "250m"
---
apiVersion: v1
kind: Service
metadata:
  name: microsvcmonirul
  namespace: monirul
  labels:
    run: microsvcmonirul
 spec: ports:
  - port: 8087
    targetPort: 8080
  selector:
    run: microsvcmonirul

Now, run the following command to deploy the microservice into the Kubernetes cluster:
[root@kmaster microservice]# kubectl apply -f deployment.yml
Once complete, the new pod will start up in the cluster as shown below-

To access the microservice's operational activity, expose the service ports to the public ip,

[root@kmaster microservice]# kubectl patch svc microsvcmonirul -n monirul -p '{"spec": {"type": "LoadBalancer", "externalIPs":["149.20.184.84"]}}'

If I try to access the Golang REST API from my browser, it will return the expected results below seen-

Now that the application is running as a microservice in a Kubernetes cluster, let's auto scale my application horizontally in response to a sudden increase or decrease in resource demand

Horizontal Pod Autoscaler (HPA)

The HPA scales the number of pods in a deployment based on a custom metric or a resource metric of a pod. Kubernetes admins can also use it to set thresholds that trigger autoscaling through changes to the number of pod replicas inside a deployment controller.

For example, if there is a sustained spike in CPU utilization above a designated threshold, the HPA will increase the number of pods in the deployment to manage the new load to maintain smooth application function.

To configure the HPA controller to manage a workload, create a HorizontalPodAutoscaler object. Or, HPA can also be configured with the kubectl autoscale subcommand. Here I’m going to use subcommand-

 apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: microsvcmonirul
  namespace: monirul
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: microsvcmonirul
  targetCPUUtilizationPercentage: 50

Using the following subcommand to create an autoscaling CPU deployment,

[root@kmaster microservice]# kubectl autoscale deployment microsvcmonirul -n monirul --cpu-percent=50 --min=1 --max=4

This will increase pods to a maximum of four replicas when the microservice deployment observes more than 50% CPU utilization over a sustained period.

To check the HPA status with namespace monirul, run the kubectl get hpa -n monirul command, which will give us the current and target CPU consumption. Initially an ''unknown'' value can appear in the current state, but with time to pull metrics, the server and percentage utilization will start to appear.

For a detailed HPA status, use the describe command to find details such as metrics, events and conditions.

kubectl describe hpa -n monirul

The above microservice running in a single pod has less than 50% CPU utilization, there is no need to auto scale the pods.

Trigger microservice autoscaling by applying load

To introduce load on the application, we use a BusyBox image in a container, which will run a shell script to make infinite calls to the REST endpoint created in the previous microservice. BusyBox is a lightweight image of many common Unix utilities -- like wget -- which we use to put stress on the microservice. This stress increases the resource consumption on the pods.

Save the following YAML configuration to a file named infinite-calls-monirul.yaml. At the bottom of the code, the wget command calls the REST API on an infinite while loop.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: infinite-calls-monirul
  namespace: monirul
  labels:

    app: infinite-calls-monirul
spec:
  replicas: 1
  selector:
    matchLabels:
      app: infinite-calls-monirul
template:
 done"
metadata:
  name: infinite-calls-monirul
  labels:
    app: infinite-calls-monirul
spec:
  containers:
  - name: infinite-calls-monirul
 image: busybox
command:
- /bin/sh
- -c
- "while true; do wget -q -O- http://149.20.184.84:8087/employee;

Deploy this YAML configuration with the kubectl apply -f infinite-calls-monirul.yml command.

Once the container is active, run a /bin/sh shell on the container using the kubectl exec -it <CONTAINER_NAME> sh command to verify that a process is running and performing web requests to the REST endpoint infinitely. These infinite calls introduce load on the application and result in processor time on the container hosting this web application.

After a few minutes of running under this load, the HPA begins to observe an increase in current CPU utilization and auto scales to manage the incoming load. It creates the maximum number of pods to maintain CPU below that 50% -- that is why the replica count is now four, which is the maximum.
kubectl get hpa -w -n monirul

To see the detailed events and activity of the HPA, run the following command and observe the highlighted section below for the events and autoscaling triggers.
kubectl describe hpa -n monirul

Vertical Pod Autoscaler

The VPA increases and decreases the CPU and memory resource requests of pod containers to better match the allocated cluster resource to actual usage. Container resource limits are based on live metrics from a metric server, rather than manual adjustments to benchmark resource utilization on the pods.

In other words, a VPA frees users from manually setting up resource limits and requests for the containers in their pods to match the current resource requirements.

The VPA can only replace the pods managed by a replication controller, such as deployments, and it requires the Kubernetes metrics server.

A VPA has three main components:

Recommender: Monitors resource utilization and computes target values. In the recommendation mode, VPA will update the suggested values but will not terminate pods.
Updater: Terminates the pods that were scaled with new resource limits. Because Kubernetes can't change the resource limits of a running pod, VPA terminates the pods with outdated limits and replaces them with pods with updated resource request and limit values.
Admission Controller: Intercepts pod creation requests. If the pod is matched by a VPA config with mode not set to "off," the controller rewrites the request by applying recommended resources to the pod specification.

Conflicts, caveats and challenges in autoscaling

Kubernetes autoscaling demonstrates flexibility and a powerful use case: It dynamically manages infrastructure scaling in production environments and enhances resource utilization, which reduces overhead.

HPA and VPA are useful, and there is a temptation to use both, but this can lead to potential conflicts. For example, HPA and VPA detect CPU at threshold levels. And while the VPA will try to terminate the resource and create a new one with updated thresholds, HPA will try to create new pods with old specs.

This can lead to wrong resource allocations and conflicts.
To prevent such a situation and still use HPA and VPA in parallel, make sure they rely on different metrics to auto scale.