Forem: Woobuntu

How to Enable Google OIDC Login in Vault Using Helm and Terraform

Woobuntu — Wed, 16 Jul 2025 07:19:27 +0000

This post documents a real-world example of enabling Google OIDC-based login for Vault.

It is based on an environment where Vault is managed via the official Helm chart using Terraform.

If you're looking to integrate a unified authentication mechanism across your organization using OIDC, this post may help.

For better understanding, check out this article first:

OAuth 2.0 and OpenID Connect

Why we introduced OIDC

Every employee in our organization is issued a Google Workspace account. As we adopted internal tools like Vault, ArgoCD, Grafana, and Jenkins, managing access control became increasingly important.

Creating and managing separate user accounts for each tool was both tedious and insecure. So we decided to unify authentication using Google OIDC, leveraging the accounts already in place.

This greatly reduced the complexity of user management and brought consistency to authentication across all systems.

Setup Process

Reference:
hashicorp/vault-guides - oidc-auth

1. Set up an OAuth client in Google Cloud Console

This process could potentially be replaced with the Terraform resource google_iam_oauth_client in the future:

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/iam_oauth_client

I plan to refactor accordingly.

Go to Google Cloud Console
Select your project
Navigate to APIs & Services → Credentials
Click Create Credentials → OAuth client ID
Choose Web application as the application type
Select Web application as the Application type
Set a name
Add your Vault origin in Authorized JavaScript origins

Set the redirect URI as defined in the official documentation:

https://<your-vault-domain>/ui/vault/auth/oidc/oidc/callback

Once created, securely store the client_id and client_secret (e.g., Vault, AWS Secrets Manager)

2. Configure OIDC in Vault

// vault-config.tf
resource "vault_auth_backend" "oidc" {
  type = "oidc"
  path = "oidc"
}

resource "vault_generic_endpoint" "oidc_config" {
  path = "auth/oidc/config"

  data_json = jsonencode({
    "oidc_discovery_url" = "https://accounts.google.com"
    "oidc_client_id"     = local.vault_secrets.vault_google_client_id
    "oidc_client_secret" = local.vault_secrets.vault_google_client_secret
  })
}

// vault-users.tf
resource "vault_generic_endpoint" "woobuntu" {
  path = "auth/oidc/role/woobuntu"

  data_json = jsonencode({
    user_claim            = "email"
    oidc_scopes           = "openid email"
    bound_audiences       = [local.vault_secrets.vault_google_client_id]
    allowed_redirect_uris = ["https://vault.my-company.com/ui/vault/auth/oidc/oidc/callback"]
    policies              = [
      // your Vault policies here
    ]
    ttl = "1h"
    bound_claims = {
      "email" = "woobuntu@my-company.com"
    }
  })
}

We injected the client_id and client_secret issued in step 1. The remaining configuration follows the documentation below:

https://github.com/hashicorp/vault-guides/tree/master/identity/oidc-auth#configure-vault

https://developer.hashicorp.com/vault/api-docs/auth/jwt#create-update-role

💡 Note on role configuration

We chose to create a separate role per user because Vault does not currently support mapping multiple users to a single OIDC role via email matching. Initially, we tried assigning multiple emails via bound_claims, but bound_claims_type only supports string or glob—not arrays.

In the future, we plan to adopt group-based access control using the groups_claim and bound_claims.group features. This depends on broader adoption of Google Workspace groups within our organization.

💡 Security Tip
Avoid hardcoding sensitive credentials like client_id and client_secret in .tfvars or local variables. Instead, use a secure secret manager such as Vault or AWS Secrets Manager:

locals {
  vault_secrets = jsondecode(data.aws_secretsmanager_secret_version.vault_secrets.secret_string)
}

How to Enable Google OIDC Login in Grafana Using Helm and Terraform

Woobuntu — Wed, 16 Jul 2025 06:03:40 +0000

This post documents a real-world case of enabling Google OIDC-based login for Grafana.

It is written based on an environment where Grafana is managed via the official Helm chart using Terraform.

This guide may be helpful if you're looking to integrate a centralized authentication system into internal tools using OIDC.

For better understanding, check out this post first:

OAuth 2.0 and OpenID Connect

Why we introduced OIDC

All members in our organization are issued Google Workspace accounts. As we adopted several internal tools—Grafana, Argo CD, Vault, Jenkins, and others—user access management became increasingly important.

Managing separate user accounts for each tool was not only cumbersome but also introduced security concerns. To unify authentication across all services, we adopted Google OIDC as a centralized login mechanism.

This allowed us to simplify user management and establish a consistent authentication flow across the system.

Setup Process

Refer to the Grafana documentation:
Configure Google OAuth authentication | Grafana docs

1. Register OAuth Client in Google Cloud Console

Go to Google Cloud Console
Select your project
Navigate to APIs & Services → Credentials
Click Create Credentials → OAuth client ID
Choose Web application as the application type
Select Web application as the Application type
Set a name
Add your Grafana origin in Authorized JavaScript origins
Set the redirect URI as defined in the official documentation:
```
https://<your-grafana-domain>/login/google
```
Once created, securely store the client_id and client_secret (e.g., Vault, AWS Secrets Manager)

Configure `grafana.ini` via Helm and Terraform

We configured the grafana.ini values through Terraform like this:

// modules/grafana/main.tf
resource "helm_release" "default" {
  ...
  chart      = "grafana"
  repository = "https://grafana.github.io/helm-charts"

  values = [
    file("${path.module}/values.yaml"),
    jsonencode({
      ...
      "grafana.ini" = merge({
        ...
      }, var.grafana_ini)
      env = var.grafana_env
    })
  ]
}

// grafana.tf
module "grafana" {
  source             = "../modules/grafana"
  storage_class_name = "grafana"
  ...

  grafana_ini = {
    ...
    "auth.google" = {
      enabled         = true
      allow_sign_up   = true
      auto_login      = true
      client_id       = local.grafana_secrets.grafana_google_client_id
      client_secret   = local.grafana_secrets.grafana_google_client_secret
      scopes          = "openid profile email"
      auth_url        = "https://accounts.google.com/o/oauth2/v2/auth"
      token_url       = "https://oauth2.googleapis.com/token"
      api_url         = "https://openidconnect.googleapis.com/v1/userinfo"
      allowed_domains = "my-company.com gmail.com"
      use_pkce        = true
    }
    server = {
      root_url = "https://grafana.my-company.com"
    }
  }
}

The client_id and client_secret are injected using the values created in step 1. The other settings follow the official documentation.

💡 Security Tip: Avoid hardcoding sensitive credentials like client_id and client_secret in your .tfvars or local values. Instead, load them from a secure secret manager such as Vault or AWS Secrets Manager:

locals {
  grafana_secrets = jsondecode(data.aws_secretsmanager_secret_version.grafana_secrets.secret_string)
}

Implementing Google OIDC Login for Argo CD (Helm + Terraform Setup)

Woobuntu — Tue, 15 Jul 2025 08:13:28 +0000

This post documents a real-world example of integrating Google OIDC login into Argo CD.

It's written based on an infrastructure where the Argo CD Helm chart is managed via Terraform.

If you're looking to unify authentication across your organization using OIDC, this guide may help.

👉 For better understanding, check out this related post first:

OAuth2.0 and OpenID Connect

Three Ways to Apply Google OIDC in Argo CD

🔗 Official Docs: Google - Argo CD - Declarative GitOps CD for Kubernetes

OpenID Connect using Dex This method does not support Google Workspace group claims (i.e., you can’t authorize users based on group membership).

💡 Since our company doesn’t actively use Google Workspace groups, we chose this method.

SAML App Auth using Dex
This method is discouraged by the Dex maintainers.
OpenID Connect + Google Groups using Dex
Supports access control based on group claims (if you actively use Google Groups).

What is Dex?

🔗 Dex Connectors Documentation

Dex is an identity provider (IdP) that connects to various authentication backends such as SAML, LDAP, GitHub, and Google, and exposes a unified OpenID Connect (OIDC) interface to clients.
In other words, Dex acts as a bridge that standardizes diverse authentication sources under a single OIDC protocol.

Implementation Steps

Reference: Argo CD Docs - OpenID Connect using Dex

1. Creating OAuth Client in Google Cloud Console

This process could potentially be replaced with the Terraform resource google_iam_oauth_client in the future:

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/iam_oauth_client

I plan to refactor accordingly.

Steps:

Log in to Google Cloud Console
Select a project
Navigate to APIs & Services
Go to Credentials
Click Create credentials → OAuth client ID

Select Web application as the Application type
Set a name
Add your ArgoCD origin in Authorized JavaScript origins
Set the redirect URI as defined in the official documentation:
```
https://<argocd domain>/api/dex/callback
```
Once created, securely store the client_id and client_secret (e.g., Vault, AWS Secrets Manager)

2. Configuring `argocd-cm` via Helm + Terraform

// modules/argocd/main.tf
resource "helm_release" "default" {
  ...
  chart            = "argo-cd"
  repository       = "https://argoproj.github.io/argo-helm"
  ...

  values = [
    file("${path.module}/values.yaml"),
    yamlencode({
      ...
      configs = {
        ...
        rbac = {
          "policy.csv" = join("\n", concat([
            "p, role:terraform, repositories, *, *, allow",
            "g, terraform-user, role:terraform"
          ], var.additional_policies))
          scopes = var.rbac_scopes // This corresponds to OIDC scopes
        }
        cm = {
          "server.rbac.log.enforce.enable" = true
          ...
          "dex.config"                     = yamlencode(var.dex_config)
          url                              = "https://${local.argocd_ingress_domain}" // required for redirect URI
          ...
        }
      }
      ...
    })
  ]
}

// argocd.tf

module "argocd" {
  source                     = "../modules/argocd"
  cluster_name               = var.cluster_name
  argocd_ingress_root_domain = "my-company.com"

    ...
  // Google OIDC uses email as the username
  rbac_scopes = "[email]"

  dex_config = {
    "connectors" = [
      {
        config = {
          issuer       = "https://accounts.google.com"
          clientId     = local.argocd_secrets.argocd_google_client_id
          clientSecret = local.argocd_secrets.argocd_google_client_secret
        }
        type = "oidc"
        id   = "google"
        name = "Google"
      }
    ]
  }
  ...
}

The clientId and clientSecret were injected using the values generated in the previous step.
Other configuration values were referenced from the following documentation:

https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/google/#configure-argo-to-use-openid-connect

https://dexidp.io/docs/connectors/oidc/

💡Since clientId and clientSecret are sensitive credentials,
it is highly recommended not to hardcode them in .tfvars or local blocks. Instead, use a secret manager such as Vault or AWS Secrets Manager to securely retrieve and inject these values.

locals {
    ...
    argocd_secrets  = jsondecode(data.aws_secretsmanager_secret_version.argocd_secrets.secret_string)
    ...
}

How to Enable Google OIDC Login in Jenkins Using Helm, JCasC, and Terraform

Woobuntu — Tue, 15 Jul 2025 05:18:22 +0000

📌 This post describes a real-world case of applying Google OIDC-based login to Jenkins.

The environment assumes Jenkins is installed via Helm and managed as a Terraform resource, with configuration handled using the JCasC plugin.

If you're aiming to unify authentication across tools using OIDC in your organization, this post should be a helpful reference.

🔗 You may want to read the following post first for better context:
Understanding OAuth 2.0 and OpenID Connect

Why We Introduced OIDC Authentication

All employees in our organization are issued Google Workspace accounts, and we have multiple internal tools requiring access control — including Jenkins, ArgoCD, Vault, and Grafana.

Managing separate user accounts for each tool was both time-consuming and a potential security risk.

So we introduced Google OIDC to unify authentication based on existing Google accounts,

which allowed us to reduce the complexity of user management and apply a consistent authentication model across our systems.

Implementation Steps

Jenkins OIDC plugin: OpenId Connect Authentication

🧩 Prerequisites

Jenkins is installed via Helm; Helm charts are managed using Terraform.
Jenkins configuration is handled through the JCasC plugin.

1. Creating OAuth Client in Google Cloud Console

Reference:

https://github.com/jenkinsci/oic-auth-plugin/blob/master/docs/configuration/GOOGLE.md#provider-configuration

Steps:

Log in to Google Cloud Console
Select a project
Navigate to APIs & Services
Go to Credentials
Click Create credentials → OAuth client ID
Select Web application as the Application type
Set a name
Add your Jenkins origin in Authorized JavaScript origins
Set the redirect URI as defined in the official documentation:
```
https://<your-jenkins-domain>/securityRealm/finishLogin
```
Once created, securely store the client_id and client_secret (e.g., Vault, AWS Secrets Manager)

The screenshots above are for demonstration purposes only. In real projects, you should never expose client information — especially the Client Secret — in public.

2. Configuring the `oic-auth` Plugin

// modules/jenkins/main.tf
resource "helm_release" "default" {
  ...
  chart            = "jenkins"
  repository       = "https://charts.jenkins.io"
  ...

  values = [
    file("${path.module}/values.yaml"),
    yamlencode({
      controller = {
          ...
          installPlugins = concat([
              ...
          ], var.extra_plugins)
        ...
        JCasC = {
          securityRealm         = var.yaml_encoded_security_realm
          ...
        }
      }
      ...
    })
  ]
}

// jenkins_helm.tf
module "jenkins" {
  source                      = "../modules/jenkins"
    ...
  yaml_encoded_security_realm = yamlencode({
    oic = {
      serverConfiguration = {
        wellKnown = {
          wellKnownOpenIDConfigurationUrl = "https://accounts.google.com/.well-known/openid-configuration"
          scopesOverride                  = "openid profile email"
        }
      }
      clientId       = local.jenkins_secrets.jenkins_google_client_id
      clientSecret   = local.jenkins_secrets.jenkins_google_client_secret
      userNameField  = "email"
      emailFieldName = "email"
      pkceEnabled    = true
    }
  })

  extra_plugins = [
      ...
    "oic-auth:4.494.v6b_f419104767",
    ...
  ]
  ...
}

The clientId and clientSecret were injected from the values issued earlier,
and the rest of the config values were based on the official documentation:
https://github.com/jenkinsci/oic-auth-plugin/blob/master/docs/configuration/GOOGLE.md#jcasc

⚠️ In the documentation above, wellKnownOpenIDConfigurationUrl and scopesOverride are placed directly under oic, but this caused OIDC to fail in my case.
Placing them under serverConfiguration.wellKnown — as shown in the general configuration reference — worked correctly:
https://github.com/jenkinsci/oic-auth-plugin/blob/master/docs/configuration/README.md#jcasc-configuration-reference

🔐 Since clientId and clientSecret are sensitive credentials, they should not be hardcoded in .tfvars or locals.
It is strongly recommended to retrieve them securely from Vault or AWS Secrets Manager instead:

locals {
    ...
  jenkins_secrets = jsondecode(data.aws_secretsmanager_secret_version.jenkins_secrets.secret_string)
  ...
}

Understanding OAuth 2.0 and OpenID Connect

Woobuntu — Tue, 15 Jul 2025 01:55:52 +0000

OAuth 2.0

OAuth 2.0 is an authorization protocol that allows a client application to access a user's resources with limited permissions.

Background: Why OAuth 2.0 was introduced

In this article, the term “service” refers to the Client in the OAuth 2.0 context.

Let’s say a service (the Client) supports Google login, and it needs access to the logged-in user's contact list.

If OAuth 2.0 didn't exist, the service (Client) would have to log in to Google using the user’s (Resource Owner) own ID and password, just like the user would.

In other words, the service (Client) would have to ask for the user's password.

Obviously, this is not acceptable in real-world services.

A protocol was needed to allow the service (Client) to access the user’s Google contacts without ever knowing the user's password.

That’s the background behind the creation of OAuth 2.0.

How OAuth 2.0 works

Since only the user (Resource Owner) should know their Google password, only the user should log in to Google.

So, the service (Client) must provide an interface that lets the user log in to Google directly, and during that process, ask for permission to access the user’s Google resources.

📌 This sequence diagram is written in Mermaid syntax.

Unfortunately, dev.to does not support rendering Mermaid diagrams directly.

You can view the full visual version here on Notion.

sequenceDiagram 
    box blue FrontChannel
        participant Client
        participant Google
        participant Resource Owner
    end
    box red BackChannel
        participant Client's Server
        participant Google's Authorization Server
        participant Google's Resource Server
    end
    Client->>Resource Owner: Provide an interface like "Login with Google"
    Resource Owner ->> Google: Redirected to Google login page
    Google->>Resource Owner: Ask for consent to access resources in the given scope
    Resource Owner->>Google: Give consent
    Google->>Client: Redirect to pre-registered redirect URI or callback URL with an authorization code (OAuth 2.0 Authorization Grant)
    Client->>Client's Server: Send the authorization code received from Google to the backend server

    Client's Server-->Google's Authorization Server: Send the code along with the client secret issued by Google
    Google's Authorization Server-->Client's Server: If valid, issue an access token
    Client's Server-->Google's Resource Server: Use the access token to access user resources on Google

OpenID Connect

In OAuth 2.0, the client (Client) can gain access to a user's resources,
but it is not guaranteed to know who the user is (i.e., no identity information is provided by default).

It’s technically possible to include scopes that return some identity-related info,
but that’s more of a workaround — not a proper authentication mechanism.

OpenID Connect (OIDC) was introduced to solve this problem.
It extends OAuth 2.0 to handle user authentication properly.

How OpenID Connect works

If you include the openid scope in your OAuth 2.0 request,
Google’s authorization server will return not only an access token but also an ID token.

The ID token is a JWT (JSON Web Token) that contains identity information such as email, name, etc.
By verifying or decoding this token, the client can securely identify the user.

DevOps Interview Practice #6

Woobuntu — Sun, 15 Jun 2025 14:33:10 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

Have you ever set up CI/CD pipelines using GitHub Actions or another tool like Jenkins or ArgoCD? If so, can you briefly describe what kind of pipeline you built?

My First Answer (Raw)

I build a pipelinen that adopt jenkins for CI, argoCD for CD ar my current job.

at CI stage, when there are changes to the source code, jenkins build application image and push it to ecr. then apply it's image tag to the values.yaml file in the mirroring branch. and it has name prefixed argocd/ for original branch name. this branch is used for argocd sync.

there are some reasons that is use this kind of branch strategy.

since argocd is a gitops tool, it syncs kubernetes resources based on git. so a new commit which reflects new image tag is created, argocd carry out deployment based on it.

if image tag is written in original branch, there could be some difficulties for cooperation. all coworkers should pull every time even if there are no

meaningful changes to the source code.

so i chose to make a dedicated branch for argocd sync. when there are changes to the source code, jenkins commits image tag generated by original commit to the dedicated branch. i make the commit message to contain original branch name and original commit message, so that which commit argocd is syncing are visible.

Refined Answer (with feedback)

I built a CI/CD pipeline at my current job, using Jenkins for CI and ArgoCD for CD.

In the CI stage, when changes are made to the source code, Jenkins builds the application image and pushes it to ECR. Then, it applies the new image tag to a values.yaml file in a dedicated mirroring branch. This branch is prefixed with argocd/ followed by the original branch name, and is used exclusively for ArgoCD synchronization.

I chose this strategy because ArgoCD is a GitOps tool—it syncs Kubernetes resources based on the Git repository. So, whenever a new commit containing the updated image tag is created, ArgoCD triggers a deployment based on that commit.

If we were to update the image tag directly in the original development branch, it could disrupt collaboration, as developers would have to pull new commits even when there were no meaningful code changes.

To avoid that, a dedicated sync branch is automatically created by the CI pipeline. Jenkins commits the new image tag to this branch, including the original branch name and commit message in the commit. This makes it easy to trace which source commit ArgoCD is deploying.

DevOps Interview Practice #5

Woobuntu — Sun, 15 Jun 2025 14:27:26 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

Let’s say your production application is running on EKS, and suddenly, the application becomes unresponsive. No pods are restarting, and there are no obvious error logs. How would you start troubleshooting this issue?

My First Answer (Raw)

If there are no error logs and nothing special about metrics of resources, then there's a hig chance that traffic couldn't be routed to the application, i guess.

in such cases, it would be appropriate to check the reason from client to application.
First, you should check whether the domain resolves to the right ip using tools like dig or nslookup. it should point a load balancer, because it's the only way to expose application on eks.
if the domain points right loadbalancer. then you should check if the traffic can arrive to it. there's a chance that the traffic is blocked by security group or acl.
if you're sure that traffic can arrive to the loadbalancer, then you should check if the traffic can be forwarded from loadbalancer to a node. the security group of nodes should allow one of loadbalancer.
then you should check if the kube proxy can forward the traffic to pods. if you are using ingress, then the traffic is first routed to ingress controller, then is routed again to the appropriate service.
if there's no problem until here, then it must be pod's problem. since there are no problems of logs and metrics, it would be readiness problem, not liveness.

Refined Answer (with feedback)

If there are no error logs and nothing unusual in the resource metrics, there's a high chance that the traffic is not reaching the application.

In such cases, it's appropriate to troubleshoot from the outside in — from the client to the application.

First, I would check whether the domain resolves to the correct IP address using tools like dig or nslookup. In EKS, the domain should typically point to a load balancer, since that's the standard way to expose services externally.

If the domain is pointing to the correct load balancer, then I would verify whether traffic is actually reaching it. It's possible that the traffic is blocked by a security group or a network ACL.

If I'm sure that the traffic reaches the load balancer, the next step is to check whether the load balancer can forward the traffic to the nodes. The nodes’ security groups must allow traffic from the load balancer's security group.

After that, I would check whether the traffic is successfully routed from the node to the pods.
If you're using a LoadBalancer service, kube-proxy handles the routing directly to the pods. If you're using Ingress, then the traffic first goes to the Ingress Controller, which forwards it to the correct Service, and then to the pods.

If everything seems fine up to this point, then the issue may be at the pod level. Since there are no error logs and resource usage is normal, it’s more likely a readiness issue than a liveness one.

DevOps Interview Practice #4: What is the difference between a rolling update and a recreate strategy in Kubernetes Deployments?

Woobuntu — Fri, 06 Jun 2025 14:17:44 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

What is the difference between a rolling update and a recreate strategy in Kubernetes Deployments?

My First Answer (Raw)

Recreate is a deploymeny strategy that ensure version persistency but sacrifice availability during version update. On the other hand, Rolling Update is one that ensure availability but sacrifice version persistency during update.

Recreate strategy ends all existing pods immediately, and when it's done, then starts scale out of new replicaset.

RollingUpdate strategy ensures availability by maxUnavailable and maxSurge. existing pods are decremended under the maxUnavailable, new pods are created under the maxSurge.

Refined Answer (with feedback)

Recreate is a deployment strategy that ensures version consistency but sacrifices availability during the update. On the other hand, RollingUpdate prioritizes availability, even if that means temporarily running different versions of the application.

With the Recreate strategy, all existing pods are terminated first. Only after that process is complete does the new ReplicaSet begin scaling up.

RollingUpdate maintains availability using two key parameters: maxUnavailable and maxSurge. Existing pods are scaled down according to the maxUnavailable setting, while new pods are created within the limit defined by maxSurge.

DevOps Interview Practice #3: What happens when you scale a StatefulSet from 2 replicas to 3?

Woobuntu — Thu, 29 May 2025 13:01:22 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

What happens when you scale a StatefulSet from 2 replicas to 3?

My First Answer (Raw)

When you scale out statefulset's replicas from 2 to 3, a new pod is generated with index 2. statefulset manages pods the way it can distinguish each pod by it's name and index, a newly populated pod has expectable name.
you can access the pod with the address like 'pod name.namespace.svc cluster.local' which is offered by headless service, and this is how it can be uniquely identifiable and accesable in a cluster.
Also, if volumeclaimtemplates are defined, a pvc is generated with a new pod.
If the storage class supports dynamic provisioning, condition matching pv is automatically made and bound to the pvc. If condition matching pv already exists, it could be bound.
the pvc is uniquley linked to the pod's name so that data can be persistent even if a pod is removed and regenerated. This ensure statefulset can maintain data persistency and pod's uniqueness.

Refined Answer (with feedback)

When you scale a StatefulSet from 2 to 3 replicas, a new pod with the index 2 is created.
StatefulSet manages pods in a way that each one is uniquely identifiable by its name and index, so the newly created pod gets a predictable name.

You can access this pod using a DNS address like pod-name.namespace.svc.cluster.local, which is provided by a headless service.
This allows the pod to be uniquely addressable within the cluster.

If volumeClaimTemplates are defined, a new PVC is automatically created for the new pod.
If the associated StorageClass supports dynamic provisioning, a new PV that matches the claim’s requirements will be provisioned and bound automatically.
If a matching PV already exists, it will be used instead.

Since the PVC is uniquely associated with the pod’s name, the pod can reconnect to the same volume even after it is deleted and recreated.
This ensures data persistence and stable identity, which are key characteristics of StatefulSets.

DevOps Interview Practice #2: What’s the difference between a Deployment and a StatefulSet in Kubernetes?

Woobuntu — Thu, 29 May 2025 12:58:56 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

What’s the difference between a Deployment and a StatefulSet in Kubernetes?

My First Answer (Raw)

Deployment is a controller which manages pods that don't need to be distinguishable because they are stateless. On the other hand, Statefulset is a controller which manages pods that need to be distinguishable becauae theu are stateful. sharded database is well known example for statefulset. each pod has to mount different volume so they must be distinguished.

Refined Answer (with feedback)

A Deployment is a controller that manages pods which don’t need to be distinguishable because they are stateless.
On the other hand, a StatefulSet is used for managing pods that must be distinguishable, since they are stateful.
A sharded database is a well-known example of a workload suited for StatefulSets, as each pod needs to mount a different volume and maintain a unique identity.

DevOps Interview Practice #1: What tool do you use to check if a pod is healthy?

Woobuntu — Thu, 29 May 2025 12:55:57 +0000

I'm a Korean DevOps engineer preparing for international opportunities. Since English isn’t my first language, I’ve been practicing both my language skills and technical knowledge at the same time.

As part of my daily English practice, I asked ChatGPT (acting as my senior DevOps engineer) to give me one interview-style question each day. I try to answer in English based on what I know, and then improve my explanation through feedback and correction.

Here’s today’s question and my answer:

Question

What tool do you use to check if a pod is healthy?

My First Answer (Raw)

There are two options to check pod's health, which are readiness probe and liveness probe. First, readiness probe is literally the option that determine whether the pod can receive traffic. If it fails, the pod shouldn't receive traffic anymore, so kube proxy eliminates it's endpoint from service. Liveness probe is literally the option which check whether the pod is alive. If the pod is running but it can not behave properly for any reason, kubelet restarts it. With their nature in my mind, they should run repeateadly in a pod's lifecycle. However, It's important to note that they are mutually independent. It's possible to pass the liveness check but readiness check, or vice verca. So you should pay attention configuring liveneas check. Because it is possible for the pod can't be ready forever because the kubelet restarts the pod before readiness probe succeeds.
You can prepare this kind of accident using startup probe. It ensures if the pod is ready to accept readiness and liveness check. if the pod have long warming up process, it's useful to configure startup probe to delay readiness and liveness check.

Refined Answer (with feedback)

There are two main mechanisms to check a pod’s health: the readiness probe and the liveness probe.

The readiness probe determines whether a pod is ready to receive traffic.
If it fails, the pod is removed from the service’s endpoints by kube-proxy, so that no traffic is routed to it.

The liveness probe, on the other hand, checks whether the pod is still alive.
Even if the container is technically running, the kubelet will restart it if the liveness probe fails repeatedly—assuming it's no longer functioning properly.

Because of their nature, both probes typically run repeatedly throughout a pod's lifecycle.
It's important to note, however, that these probes operate independently:
a pod may pass one while failing the other.

This can be problematic—especially if the liveness probe is too aggressive.
In such cases, the pod might get restarted before it even has a chance to become ready, causing it to remain in a perpetual unready state.

To handle this, you can configure a startup probe, which is designed to delay the execution of readiness and liveness checks until the application has finished initializing.
For pods with a long startup time, this is an effective way to prevent premature restarts and ensure proper lifecycle management.

Forem: Woobuntu

How to Enable Google OIDC Login in Vault Using Helm and Terraform

Why we introduced OIDC

Setup Process

1. Set up an OAuth client in Google Cloud Console

2. Configure OIDC in Vault

How to Enable Google OIDC Login in Grafana Using Helm and Terraform

Why we introduced OIDC

Setup Process

1. Register OAuth Client in Google Cloud Console

Configure grafana.ini via Helm and Terraform

Implementing Google OIDC Login for Argo CD (Helm + Terraform Setup)

Three Ways to Apply Google OIDC in Argo CD

What is Dex?

Implementation Steps

1. Creating OAuth Client in Google Cloud Console

2. Configuring argocd-cm via Helm + Terraform

How to Enable Google OIDC Login in Jenkins Using Helm, JCasC, and Terraform

Why We Introduced OIDC Authentication

Implementation Steps

🧩 Prerequisites

1. Creating OAuth Client in Google Cloud Console

2. Configuring the oic-auth Plugin

Understanding OAuth 2.0 and OpenID Connect

OAuth 2.0

Background: Why OAuth 2.0 was introduced

How OAuth 2.0 works

OpenID Connect

How OpenID Connect works

DevOps Interview Practice #6

Question

My First Answer (Raw)

Refined Answer (with feedback)

DevOps Interview Practice #5

Question

My First Answer (Raw)

Refined Answer (with feedback)

DevOps Interview Practice #4: What is the difference between a rolling update and a recreate strategy in Kubernetes Deployments?

Question

My First Answer (Raw)

Refined Answer (with feedback)

DevOps Interview Practice #3: What happens when you scale a StatefulSet from 2 replicas to 3?

Question

My First Answer (Raw)

Refined Answer (with feedback)

DevOps Interview Practice #2: What’s the difference between a Deployment and a StatefulSet in Kubernetes?

Question

My First Answer (Raw)

Refined Answer (with feedback)

DevOps Interview Practice #1: What tool do you use to check if a pod is healthy?

Question

My First Answer (Raw)

Refined Answer (with feedback)

Configure `grafana.ini` via Helm and Terraform

2. Configuring `argocd-cm` via Helm + Terraform

2. Configuring the `oic-auth` Plugin