Forem: Ishan Sharma

Expanding Persistent Volume for Statefulsets in Kubernetes

Ishan Sharma — Thu, 23 Nov 2023 16:51:49 +0000

Expanding volumes in Kubernetes clusters is a routine maintenance activity and seemingly straightforward – or so I thought! Recently, a unique challenge arose when there was a request to increase the storage size for a Stateful Application running in a Kubernetes cluster.

Let's jump into the story to share the real-world experience of this request.

Problem Statement

In the realm of Kubernetes clusters, a statefulset application thrived. Its popularity soared to the extent that it demanded more storage than the currently attached volume could provide. The team, recognizing the need to meet this demand, set out to expand the storage of the persistent volume connected to the statefulset. However, quickly realized that this was easier said than done.

As kubernetes 1.24 supports volume expansion, Just like anyone who had the first encounter with this task, I anticipated that it was as simple as increasing the spec.volumeClaimTemplates.spec.resources.requests.storage in the statefulset application. But It was not !!

Contrary to initial expectations, modifying the spec.volumeClaimTemplates.spec.resources.requests.storage alone does not trigger the expansion of volumes attached to statefulsets. Manual intervention is necessary to navigate this intricate landscape. Let’s learn how can we achieve it in this article.

Prerequisites For Persistent Volume Expansion

The underlying provider and CSI driver for the persistent volume must support volume expansion. You should refer to the documentation of your CSI driver to check the capability. (Managed Kubernetes with their inbuilt CSI driver supports this natively and the feature volume expansion is also enabled)
Volume expansion must be enabled on the respective storage class used by the persistent volume. This can be checked using the below command:

## Output in Name of storageclass with support to ALLOWVOLUMEEXPANSION
$ kubectl get storageclass -o custom-columns=NAME:.metadata.name,ALLOWVOLUMEEXPANSION:.allowVolumeExpansion

Output from the above command will look something like this:

NAME                    ALLOWVOLUMEEXPANSION
azurefile               true
azurefile-csi           true
azurefile-csi-premium   true
azurefile-premium       true
default                 true
managed                 true
managed-csi             true
managed-csi-premium     true
managed-premium         true

Step-by-Step Guide With Example.

To make it more practical, I will use Alertmanager application, which is installed with kube-prometheus-stack helm chart.

Reproduce the problem

Installation of helm chart.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm upgrade --install kube-prom-stack prometheus-community/kube-prometheus-stack --namespace kube-prom-stack --create-namespace --values custom-values.yaml
Release "kube-prom-stack" does not exist. Installing it now.
NAME: kube-prom-stack
LAST DEPLOYED: Thu Nov 23 10:42:39 2023
NAMESPACE: kube-prom-stack
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace kube-prom-stack get pods -l "release=kube-prom-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Info: Application details are out of the scope of this article.

-custom-values.yaml

alertmanager:
  ingress:
    enabled: false

  podDisruptionBudget:
    enabled: false

  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi

Once the installation is complete, we can list the statefulsets.

Info: kube-prom-stack namespace is set as the default namespace.

$ kubectl get statefulsets
NAME                                                   READY   AGE
alertmanager-kube-prom-stack-kube-prome-alertmanager   1/1     11m
prometheus-kube-prom-stack-kube-prome-prometheus       1/1     11m

Check the details of Persistent volume claim created by the volumeClaimTemplates specification of the statefulset to verify if the correct values are applied.

## Get the pvc name from the statefulset volumeClaimTemplate
$ kubectl get sts alertmanager-kube-prom-stack-kube-prome-alertmanager -o jsonpath='{.spec.volumeClaimTemplates[0].metadata.name}' ; echo ""
alertmanager-kube-prom-stack-kube-prome-alertmanager-db

## Get the pvc size request from the statefulset volumeClaimTemplate
$ kubectl get sts alertmanager-kube-prom-stack-kube-prome-alertmanager -o jsonpath='{.spec.volumeClaimTemplates[0].spec.resources}' | jq
{
  "requests": {
    "storage": "2Gi"
  }
}

## Verify the size of the PVC from the statefulset volumeClaimTemplate
$ kubectl get pvc alertmanager-kube-prom-stack-kube-prome-alertmanager-db-alertmanager-kube-prom-stack-kube-prome-alertmanager-0 -o jsonpath='{.spec.resources.requests}' | jq
{
  "storage": "2Gi"
}

Storage is 2Gi which seems to be as per our provided values.yaml ✅

Extend the persistent volume claim storage request in the provided values.yaml.
Modified custom-values.yaml

alertmanager:
  ingress:
    enabled: false

  podDisruptionBudget:
    enabled: false

  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi ## Increased ##

Size is updated to 5Gi, will apply the new values and monitor the changes.

Helm upgrade with new values.

$ helm upgrade --install kube-prom-stack prometheus-community/kube-prometheus-stack --namespace kube-prom-stack --values custom-values.yaml
Release "kube-prom-stack" has been upgraded. Happy Helming!
NAME: kube-prom-stack
LAST DEPLOYED: Thu Nov 23 10:55:25 2023
NAMESPACE: kube-prom-stack
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace kube-prom-stack get pods -l "release=kube-prom-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Check the details of the Persistent volume claim after updating the size.

## Get the pvc name from the statefulset volumeClaimTemplate
$ kubectl get sts alertmanager-kube-prom-stack-kube-prome-alertmanager -o jsonpath='{.spec.volumeClaimTemplates[0].metadata.name}' ; echo ""
alertmanager-kube-prom-stack-kube-prome-alertmanager-db

## Get the PVC size request from the statefulset volumeClaimTemplate (UPDATED)
$ kubectl get sts alertmanager-kube-prom-stack-kube-prome-alertmanager -o jsonpath='{.spec.volumeClaimTemplates[0].spec.resources}' | jq
{
  "requests": {
    "storage": "5Gi"
  }
}

## Verify the size of the pvc got from the statefulset volumeClaimTemplate (NOT-UPDATED)
$ kubectl get pvc alertmanager-kube-prom-stack-kube-prome-alertmanager-db-alertmanager-kube-prom-stack-kube-prome-alertmanager-0 -o jsonpath='{.spec.resources.requests}' | jq
{
  "storage": "2Gi"
}

Even though the volumeClaimTemplate of statefulset(alertmanager) is updated yet the persistent volume claim is using the old size 2Gi.

Solution

The following steps are required to increase the size from 2Gi to 5Gi.

STEP 1: In this case the statefulset is managed via the operator hence needs to be paused as well on the custom resource alertmanager

Get the name of alertmanager resource.

$ kubectl get alertmanager
NAME                                      VERSION   REPLICAS   READY   RECONCILED   AVAILABLE   AGE
kube-prom-stack-kube-prome-alertmanager   v0.26.0   1          1       True         True        61m

Patch the resource so that the operator does not recreate it.

$ kubectl patch alertmanager/kube-prom-stack-kube-prome-alertmanager  --patch '{"spec": {"paused": true, "storage": {"volumeClaimTemplate": {"spec": {"resources": {"requests": {"storage":"5Gi"}}}}}}}' --type merge
alertmanager.monitoring.coreos.com/kube-prom-stack-kube-prome-alertmanager patched

STEP 2: Patch the persistent volume claims associated with the statefulset.

for p in $(kubectl get pvc -l alertmanager=kube-prom-stack-kube-prome-alertmanager -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \
  kubectl patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"5Gi"}}}}'; \
done

STEP 3: Final step is to delete the statefulset with the orphan strategy.

By passing --cascade=orphan to kubectl delete, the Pods managed by the StatefulSet are left behind even after the StatefulSet object itself is deleted.

$ kubectl delete sts alertmanager-kube-prom-stack-kube-prome-alertmanager --cascade=orphan

To verify if the PVC is extended.

## PVC is updated
$ kubectl get pvc alertmanager-kube-prom-stack-kube-prome-alertmanager-db-alertmanager-kube-prom-stack-kube-prome-alertmanager-0 -o jsonpath='{.spec.resources.requests}' | jq
{
  "storage": "5Gi"
}

STEP 4: Resume the operator to take control of the alertmanager provisioning. This step is valid in statefultsets managed via operators only.

kubectl patch alertmanager/kube-prom-stack-kube-prome-alertmanager --patch '{"spec": {"paused": false}}' --type merge

Conclusion

In Summary following steps need to be followed in the respective order to expand the persistent volume on statefulsets.

Patch the statefulset with the required(expanded) volumeClaimTemplates configuration.
Patch the persistent volume claims with the required(expanded) size configuration associated with the statefulset.
Delete the statefulset with cascade=orphan strategy.

And that concludes our exploration into the dynamic world of scaling storage for stateful applications in Kubernetes.

Expanding the persistent volumes in statefulsets is a well-known restriction and is been tracked in the Support Volume Expansion Through StatefulSets and StatefulSet: support resize PVC storage in K8s v1.11 GitHub issues in official Kubernetes repositories.

Acknowledgments

Thank you for embarking on this Kubernetes adventure with me. Scaling storage in Kubernetes can be complex, but you've gained valuable insights to navigate this terrain. May your clusters be resilient, and your applications thrive. Happy reading, and until our next exploration!

Helpful Links

Terraform and Ansible: Teaming Up for Automated Cloud Magic

Ishan Sharma — Thu, 17 Aug 2023 15:44:04 +0000

TL;DR

In the world of cloud automation, Terraform and Ansible form a seamless partnership. Terraform constructs infrastructure, while Ansible configures it. Leveraging dynamic inventories and GitHub Actions, the process gains efficiency.

Explore my GitHub repository for hands-on experience. Delve into Terraform's provisioning, Ansible's management, and GitHub's orchestration.

Unlock cloud automation with Terraform, Ansible, and GitHub's synergy.

Introduction

In the dynamic world of cloud computing, provisioning infrastructure and managing configurations are essential tasks. This is where Terraform and Ansible come into play, acting as a dynamic duo that enables you to orchestrate automated cloud magic.

The Power of Terraform and Ansible

Terraform: Infrastructure Provisioning Simplified

Terraform stands as a powerful infrastructure-as-code tool. It allows you to define your cloud infrastructure using a human-readable syntax. This approach streamlines the process of spinning up resources on cloud platforms like Azure. Whether it's virtual machines, networking components, or databases, Terraform's declarative approach ensures consistent provisioning across environments.

Ansible: Configuration Management Perfected

On the other hand, Ansible specializes in configuration management. It allows you to define the desired state of your servers and applications. Ansible playbooks, written in simple YAML syntax, automate the process of configuring servers, installing software, and ensuring consistency across your infrastructure. This comes in handy when you're dealing with tasks like setting up web servers or managing security configurations.

Complementary, Not Competitive

Dispelling the Misconception

One common misconception is that Terraform and Ansible compete with each other. In reality, they are highly complementary. Terraform focuses on creating and destroying resources, while Ansible excels in configuring and maintaining those resources. This synergy ensures that your infrastructure is not just provisioned but also tailored to meet your specific requirements.

Dynamic Inventories and Pipeline Automation

Dynamic Inventories: A Game Changer

A remarkable feature that enhances this collaboration is the use of dynamic inventories. Instead of maintaining static inventory lists, Ansible can directly fetch information about your cloud resources from the likes of Azure using dynamic inventory plugins. This makes your playbooks flexible and adaptable to the evolving cloud landscape.

Seamless Automation with GitHub Actions

Bringing it all together, GitHub Actions empowers you to automate your workflows. With GitHub as your source version control, you can leverage GitHub Actions to define pipelines that seamlessly integrate Terraform and Ansible. Pushing code triggers the orchestration of provisioning infrastructure and configuring it, all without manual intervention.

Embarking on Practical Cloud Journey

Hands-On Learning

For those eager to dive into practical knowledge, there's a treasure trove awaiting you. Inside the GitHub repository, you'll find a rich collection of code that practically demonstrates the synergy between Terraform and Ansible. Each line of code showcases how to orchestrate cloud resources and configure them seamlessly.

ishuar / terraform-ansible-azure

Terraform and Ansible: Teaming Up for Automated Azure Cloud Magic

Terraform and Ansible Hand In Hand

🌩️ Terraform and Ansible: Teaming Up for Automated Cloud Magic 🌩️
Report Bug or Request Feature

Introduction

Terraform and Ansible 🤝 are powerful tools that can work synergistically to provision and configure cloud infrastructure. In this repository, we'll explore how to utilize Terraform for infrastructure provisioning and Ansible for configuration management, all within the context of Microsoft Azure.

Prerequisites

Before diving into using Terraform and Ansible for your Azure cloud infrastructure, ensure you have the following prerequisites in place:

Prerequisite	Description
Azure Account	You must have a valid Azure account to create and manage resources on the Azure cloud platform.
Terraform Installed	Install

…

View on GitHub

By immersing yourself in this repository, you're not just reading about automation; you're experiencing it firsthand. Through tinkering, testing, and exploration, you'll uncover the magic that comes to life when Terraform and Ansible work in harmony.

A glimpse of the tools and components involved is as shown below

Terraform as your spell book 📓

First thing first, please refer to Readme file within the terraform/linux-webserver-with-loadbalancer directory for pre-requisites to replicate the infrastructure on your local environment.

INFO: For best experience open all embedded links in a new browser window/tab 💻.

Ansible as your ancient scroll 📜

First thing first, please refer to Readme file within the ansible directory for pre-requisites to replicate the infrastructure on your local environment.

Ansible Configuration.


##? Generate complete using: ansible-config init --disabled -t all > <path>/ansible.cfg

[defaults]

# (boolean) Set this to "False" if you want to avoid host key checking by the underlying tools Ansible uses to connect to the host
host_key_checking = False
force_color = True

# (integer) Port to use in remote connections, when blank it will use the connection plugin default.
## As we have changed the default SSH port of our VMs
remote_port=8822

[privilege_escalation]

# (boolean) Toggle to prompt for privilege escalation password.
become_ask_pass=False

# (string) Privilege escalation method to use when `become` is enabled.
become_method=sudo

# (string) The user your login/remote user 'becomes' when using privilege escalation, most systems will use 'root' when no user is specified.
become_user=root

Dynamic Inventory

Refer to pre-requisites for local set environment set up.

---
plugin: azure_rm

include_vm_resource_groups:
  - ansible-vm-resources

auth_source: auto
conditional_groups:
  # since this will be true for every host, every host sourced from this inventory plugin config will be in the
  # group 'all_the_hosts'
  all_the_hosts: true

# places hosts in dynamically-created groups based on a variable value.
keyed_groups:
  # places each host in a group named 'tag_(tag name)_(tag value)' for each tag on a VM.
  # - prefix: tag
  #   key: tags
  # places each host in a group named 'azure_loc_(location name)', depending on the VM's location
  - prefix: azure_loc
    key: location
  # places host in a group named 'some_tag_X' using the value of the 'sometag' tag on a VM as X, and defaulting to the
  # value 'none' (eg, the group 'some_tag_none') if the 'sometag' tag is not defined for a VM.
  - prefix: role
    key: tags.role | default('none')

Ansible Roles Involved
Ansible Playbook

---
- name: Set up Nginx Webserver on Ubuntu machine
  gather_facts: true
  remote_user: adminuser
  hosts: "{{ dynamic_hosts }}"
  become: true
  connection: ssh
  pre_tasks: []
  vars:
    dynamic_hosts: role_slave_webservers

  roles:
    - role: azure_vm_ufw
      when: enable_firewall | bool
    - role: nginx_webserver

Realm of GitHub Actions 🪐

Concept of Github reusable workflows are utilised in the repository, hence create workflow one time and then can re-use it for supporting used case.

Reusable Workflows

Deployment and Configuration Workflows

webservers-infra-terraform.yaml

name: "Create Webservers Infrastructure"
on:
  workflow_dispatch:
    inputs:
      terraform-version:
        type: number
        required: false
        default: 1.5.4
        description: The terraform version used for the github action.

      cache-hash-file:
        type: string
        required: false
        default: '/providers.tf'
        description: The file used to create common hash cache naming.
  push:
      branches:
        - main
      paths:
      - "terraform/**"
      - ".github/workflows/terraform-infra-set-up.yaml"
      - ".github/workflows/webservers-infra-terraform.yaml"

  pull_request:
    paths:
    - "terraform/**"
    - ".github/workflows/terraform-infra-set-up.yaml"
    - ".github/workflows/webservers-infra-terraform.yaml"

concurrency:
  group: terraform-webservers

jobs:
  webserversInfra:
    name: Create infrastructure for webservers
    uses: ./.github/workflows/terraform-infra-set-up.yaml
    with:
      terraform-dir: "terraform/linux-webserver-with-loadbalancer"
      terraform-version: ${{ inputs.terraform-version != '' && inputs.terraform-version || vars.TERRAFORM_VERSION }}
    secrets: inherit

webservers-config-ansible.yaml

name: "Configure Nginx Webservers in Ubuntu via Ansible"
on:
  workflow_dispatch:
  push:
    branches:
      - main
    ## in Case push to main by codeowners
    paths:
    - "ansible/**"
    - ".github/workflows/set-up-ubuntu-nginx-webserver.yaml"
    - ".github/workflows/ansible-set-up.yaml"

  pull_request:
    paths:
    - "ansible/**"
    - ".github/workflows/set-up-ubuntu-nginx-webserver.yaml"
    - ".github/workflows/ansible-set-up.yaml"

concurrency:
  group: ansible-webservers

jobs:
  webserversConfig:
    name: Configure Nginx webservers
    uses: ./.github/workflows/ansible-set-up.yaml
    with:
      playbook: set-up-ubuntu-nginx-webserver.yaml
      terraform-output-directory: terraform/linux-webserver-with-loadbalancer
      nsg-ssh-port: 8822
    secrets:
      ssh-private-key: ${{ secrets.PASSWORDLESS_SSH_PRIVATE_KEY }}
      AZURE_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      AZURE_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      AZURE_CLIENT_SECRET: ${{ secrets.ARM_CLIENT_SECRET }}
      AZURE_SUBSCRIPTION_ID: ${{ vars.ARM_SUBSCRIPTION_ID }}

Wrapping Up

Terraform and Ansible are more than just tools; they represent a collaborative approach to cloud automation. By using Terraform for infrastructure provisioning and Ansible for configuration management, you unlock a potent synergy that ensures your cloud environment is both robust and adaptable. Add dynamic inventories and GitHub Actions into the mix, and you have a recipe for automated cloud magic that simplifies and streamlines your operations.

Happy automating!

Thank You 🙏

A heartfelt thank you and grateful for the time you've spent for reading the article. I hope I was able to ignite your curiosity and guided you through the realm of Terraform, Ansible, and cloud magic.

Your thoughts matter!

If this journey sparked ideas or questions, I'd love to hear from you. Share your feedback, suggestions via GitHub Issue or even a magical star ⭐️ for the project on GitHub.

Stay tuned for more tech insights.

ishuar / terraform-ansible-azure

Terraform and Ansible: Teaming Up for Automated Azure Cloud Magic

Terraform and Ansible Hand In Hand

🌩️ Terraform and Ansible: Teaming Up for Automated Cloud Magic 🌩️
Report Bug or Request Feature

Introduction

Prerequisites

Before diving into using Terraform and Ansible for your Azure cloud infrastructure, ensure you have the following prerequisites in place:

Prerequisite	Description
Azure Account	You must have a valid Azure account to create and manage resources on the Azure cloud platform.
Terraform Installed	Install

…

View on GitHub

Guess what?

This document got a little boost from AI magic, making it even more exciting. But don't worry, it's still your friendly guide, here to help you on your learning journey