Forem: James Maina

Getting started on MOCO, the MySQL Operator for Kubernetes Part 1

James Maina — Tue, 14 Jan 2025 05:59:49 +0000

MOCO (MySQL Operator for Kubernetes) is a robust, cloud-native solution designed to simplify the management of MySQL clusters in Kubernetes environments. It automates the provisioning, scaling, backup, and maintenance of MySQL instances while ensuring high availability and reliability. MOCO leverages Kubernetes resources to create, monitor, and manage MySQL clusters.

MOCO supports specific versions of MySQL and Kubernetes. As of the latest information, it supports MySQL versions 8.0.28, 8.0.37, 8.0.39, 8.0.40, and 8.4.3, and Kubernetes versions 1.29, 1.30, and 1.31

How MOCO Works

Provisioning Clusters

MOCO provisions MySQL clusters by creating Kubernetes StatefulSets for each cluster. The process involves:
Defining a MySQLCluster custom resource (CR) with desired configurations.
MOCO controller creates StatefulSets and persistent volume claims (PVCs) for the cluster nodes.
Configuring MySQL instances with semi-synchronous replication for high availability.

Deployment Services

MOCO deploys the following services:

Primary Service: Routes traffic to the primary node.
Replica Service: Routes traffic to replica nodes for read queries.
Backup Service: Handles backups through sidecars integrated into the StatefulSet.

Types of Deployment

Single Primary with Replicas: Default mode with one primary and multiple replicas.
Multi-Region Clusters: For cross-region replication.

Backup and restore

MOCO can take full and incremental backups regularly. The backup data are stored in Amazon S3 compatible object storages.
MOCO supports:

Scheduled backups to object storage (e.g., S3).
On-demand backups triggered through the Kubernetes API.
Restorations from backups via simple CR updates.
Point-in-Time Recovery: Ensures robust data protection in disaster recovery scenarios.

Object storage bucket
Bucket is a management unit of objects in S3. MOCO stores backups in a specified bucket.

MOCO does not remove backups. To remove old backups automatically, you can set a lifecycle configuration to the bucket.

Quick setup

You can choose between two installation methods.

MOCO depends on cert-manager. If cert-manager is not installed on your cluster, install it as follows:

Install using raw manifests:
$ curl -fsLO https://github.com/cybozu-go/moco/releases/latest/download/moco.yaml $ kubectl apply -f moco.yaml

Install using Helm chart:
$ helm repo add moco https://cybozu-go.github.io/moco/ $ helm repo update $ helm install --create-namespace --namespace moco-system moco moco/moco

Customize manifests
If you want to edit the manifest, config/ directory contains the source YAML for kustomize.

Creating a Cluster

An empty cluster always has a writable instance called the primary. All other instances are called replicas. Replicas are read-only and replicate data from the primary.

The following YAML is to create a three-instance cluster. It has an anti-affinity for Pods so that all instances will be scheduled to different Nodes. It also sets the limits for memory and CPU to make the Pod Guaranteed.

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  namespace: default
  name: test
spec:
  # replicas is the number of mysqld Pods.  The default is 1.
  replicas: 3
  podTemplate:
    spec:
      # Make the data directory writable. If moco-init fails with "Permission denied", uncomment the following settings.
      # securityContext:
      #   fsGroup: 10000
      #   fsGroupChangePolicy: "OnRootMismatch"  # available since k8s 1.20
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - mysql
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - test
            topologyKey: "kubernetes.io/hostname"
      containers:
      # At least a container named "mysqld" must be defined.
      - name: mysqld
        image: ghcr.io/cybozu-go/moco/mysql:8.4.3
        # By limiting CPU and memory, Pods will have Guaranteed QoS class.
        # requests can be omitted; it will be set to the same value as limits.
        resources:
          limits:
            cpu: "10"
            memory: "10Gi"
  volumeClaimTemplates:
  # At least a PVC named "mysql-data" must be defined.
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

By default, MOCO uses preferredDuringSchedulingIgnoredDuringExecution to prevent Pods from being placed on the same Node.
There are other example manifests in thier examples directory.

Using the cluster

kubectl moco
From outside of your Kubernetes cluster, you can access MOCO MySQL instances using kubectl-moco. kubectl-moco is a plugin for kubectl. Pre-built binaries are available on GitHub releases.

The following is an example to run mysql command interactively to access the primary instance of test MySQLCluster in foo namespace.

$ kubectl moco -n foo mysql -it test

Connecting to mysqld over network
MOCO prepares two Services for each MySQLCluster. For example, a MySQLCluster named test in foo Namespace has the following Services.

Service Name DNS Name Description moco-test-primary moco-test-primary.foo.svc Connect to the primary instance. moco-test-replica moco-test-replica.foo.svc Connect to replica instances.

Cluster status

You can see the health and availability status of MySQLCluster as follows:

$ kubectl get mysqlcluster NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS test True True 0 3

The cluster is available when the primary Pod is running and ready.
The cluster is healthy when there is no problems.
PRIMARY is the index of the current primary instance Pod.
SYNCED REPLICAS is the number of ready Pods.
ERRANT REPLICAS is the number of instances having errant transactions. You can also use kubectl describe mysqlclusterto see the recent events on the cluster

Logs

Error logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 mysqld
Slow logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 slow-log

Switchover
Switchover is an operation to change the live primary to one of the replicas.

MOCO automatically switch the primary when the Pod of the primary instance is to be deleted.

Users can manually trigger a switchover with kubectl moco switchover CLUSTER_NAME.

Failover
Failover is an operation to replace the dead primary with the most advanced replica. MOCO automatically does this as soon as it detects that the primary is down.

The most advanced replica is a replica who has retrieved the most up-to-date transaction from the dead primary. Since MOCO configures loss-less semi-synchronous replication, the failover is guaranteed not to lose any user data.

Re-initializing an errant replica
Delete the PVC and Pod of the errant replica, like this:

$ kubectl delete --wait=false pvc mysql-data-moco-test-0 $ kubectl delete --grace-period=1 pods moco-test-0
Depending on your Kubernetes version, StatefulSet controller may create a pending Pod before PVC gets deleted. Delete such pending Pods until PVC is actually removed.

Ref - https://cybozu-go.github.io/moco/index.html

What is Kured (KUbernetes REboot Daemon) in k8s?

James Maina — Fri, 01 Mar 2024 11:34:08 +0000

Defination from the official page states that kured is a Kubernetes daemonset that performs safe automatic node reboots when the need to do so is indicated by the package management system of the underlying OS.

By periodically rebooting nodes, Kured ensures that any pending updates or configuration changes take effect, resulting in a more efficient and reliable cluster.

Here are the key points to note:

Kured monitors the operating system for security patches, kernel updates, and system-level changes in Kubernetes nodes. It proactively identifies the need for reboots to keep the cluster secure and up-to-date.
When a reboot is required, Kured gracefully cordons the node, marking it as unschedulable for new pods without disrupting existing ones. It then proceeds to drain the node, evicting existing pods in a controlled manner to ensure a smooth reboot process.
Kured includes built-in safety mechanisms to prevent unnecessary reboots and allows users to define maintenance windows for avoiding disruptions.

The continuous monitoring by Kured ensures that the Kubernetes cluster operates with the latest updates, enhancing performance, security, and stability.
Organizations can leverage Kubernetes clusters more effectively while minimizing risks associated with outdated software and configurations.

Setting up Kured is a straightforward process that involves deploying it as a DaemonSet in the Kubernetes cluster. This deployment strategy ensures that Kured runs on every node within the cluster, effectively monitoring and managing the rebooting process for each individual node.

Here is how you can do that

# ClusterRole for kured
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kured
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "delete", "get"]
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]

# ClusterRoleBinding for kured
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kured
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kured
subjects:
- kind: ServiceAccount
  name: kured
  namespace: kube-system

# Role for kured in kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: kube-system
  name: kured
rules:
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  resourceNames: ["kured"]
  verbs: ["update"]

# RoleBinding for kured in kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-system
  name: kured
subjects:
- kind: ServiceAccount
  namespace: kube-system
  name: kured
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kured

# ServiceAccount for kured
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kured
  namespace: kube-system

# DaemonSet for kured
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kured
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: kured
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: kured
    spec:
      serviceAccountName: kured
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: "node-role.kubernetes.io/mysql"
        operator: "Equal"
        effect: "NoSchedule"
      hostPID: true
      restartPolicy: Always
      containers:
      - name: kured
        image: ghcr.io/kubereboot/kured:{{ kured_version }}
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        env:
        - name: KURED_NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        command:
        - /usr/bin/kured
        - --reboot-days=mon,tue,wed,thu
        - --reboot-delay=90s
        - --start-time=3am
        - --end-time=5am
        - --time-zone=UTC
        - --prometheus-url={{ prometheus_url }}
        - --alert-filter-regexp=^Watchdog$
        - --period=15m

Explanation:
This part allows you to define maintenance windows for avoiding disruptions.

  command:
        - /usr/bin/kured
        - --reboot-days=mon,tue,wed,thu
        - --reboot-delay=90s
        - --start-time=3am
        - --end-time=5am
        - --time-zone=UTC
        - --prometheus-url={{ prometheus_url }}
        - --alert-filter-regexp=^Watchdog$
        - --period=15m

Karpenter: The Better Autoscaling Solution for Kubernetes- Part 1

James Maina — Sat, 25 Feb 2023 04:33:48 +0000

If you're running Kubernetes, you're likely familiar with the standard cluster autoscaler. While it's a useful tool, it has its limitations. Enter Karpenter, an open-source autoscaling solution that offers many advantages over the standard cluster autoscaler. It is a flexible, high-performance Kubernetes cluster autoscaler built with AWS.

How Karpenter Works

Karpenter takes a different approach to autoscaling than the standard cluster autoscaler. Instead of adding or removing nodes based on demand, Karpenter provisions nodes based on application requirements. This means that it can optimize resource utilization and reduce costs.

Karpenter works by creating custom Kubernetes resources called "provisioners." Provisioners are used to define the resources that Karpenter should provision, such as nodes or virtual machines. When an application needs more resources, Karpenter checks the provisioners to see if any need to be created. If so, Karpenter will create the new resources and add them to the cluster.

Image Courtesy.

Comparing Karpenter to Cluster Autoscaler

Karpenter offers several advantages over the standard cluster autoscaler as detailed below.

Optimal Resource Utilization

One of the most significant advantages of Karpenter is its ability to optimize resource utilization. It can do this by automatically provisioning nodes based on application needs. This means that you can avoid overprovisioning, which can lead to wasted resources and increased costs.

Customizable Scaling

Karpenter offers the ability to customize scaling behaviors based on your specific needs. You can configure scaling based on metrics such as CPU or memory usage, or you can use your own custom metrics.

Cost Savings

Because Karpenter optimizes resource utilization, it can lead to significant cost savings. By avoiding overprovisioning, you can reduce the number of nodes required to run your applications, which can result in lower cloud bills.

Ease of Use

Karpenter is easy to use and deploy. It can be installed using Helm, and it integrates seamlessly with Kubernetes.

The standard cluster autoscaler can be more challenging to set up, and it may require more manual configuration.

How to Get Started with Karpenter

There are different ways to get started with Karpenter. This article will just highlight the steps. (Watch out for part 2 with a step-by-step guide on how to install and configure Karpenter) We will use Helm Chart to install Karpernter. Here are some steps before having it operational:

Create the KarpenterNode IAM Role - Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking.
Create the IAM role for Karpenter Controller - Associate the Kubernetes Service Account and the IAM role using IRSA
Update aws-auth ConfigMap - to allow the nodes that use the KarpenterRole IAM Role to join the cluster
Deploy Karpenter Helm Chart:

helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION}

Create a default Provisioner, (example)

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    intent: apps
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values: [nano, micro, small, medium, large]
  limits:
    resources:
      cpu: 1000
      memory: 1000Gi
  ttlSecondsAfterEmpty: 30
  ttlSecondsUntilExpired: 2592000
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  tags:
    KarpenerProvisionerName: "default"
    NodeType: "karpenter-workshop"
    IntentLabel: "apps"
EOF

Once you've installed Karpenter, you can begin using it to optimize your Kubernetes cluster's resource utilization and reduce costs.

Provisioner configuration

Karpenter configuration comes in the form of a Provisioner CRD (Custom Resource Definition). A single Karpenter provisioner is capable of handling many different pod shapes. You can play with it to suit whatever needs you have. For example

One can limit Karpenter to use either on-demand or spot instances, you can use the spot field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    type: karpenter
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
        # - key: karpenter.sh/capacity-type
    #   operator: In
    #   values: ["spot"]
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["c5.large", "m5.large", "m5.xlarge"]

In this example we're setting the karpenter.sh/capacity-typeto initially limit Karpenter to provisioning On-Demand instances, and karpenter.k8s.aws/instance-typeto limit to specific instance types.

One can also limit Karpenter to specific instance types, regions and zones you can use the instanceTypes field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha1
kind: Provisioner
metadata:
  name: example-provisioner
spec:
  constraints:
    - type: "awsec2"
      region: "us-west-2"
      zones:
        - "a"
      instanceTypes:
        - "t2.micro"

In this example, the instanceTypes field is set to t2.micro, which means that the provisioner will only use t2.micro instances. You can add additional instance types to the list if you want to allow for more flexibility.

Limitations

While Karpenter offers many advantages over the standard cluster autoscaler, it also has some limitations. Some of the key limitations include:

Limited support for custom metrics: While Karpenter does offer support for custom metrics, it is more limited than some other solutions. This can make it challenging to implement certain types of custom scaling behaviors.
Lack of integration with some cloud providers: Karpenter is designed to work with Kubernetes, but it may not integrate seamlessly with all cloud providers. This can make it more challenging to deploy Karpenter in certain environments.
Complexity: Karpenter is a powerful tool, but it can also be complex to configure and use. It may require more expertise and resources than some other scaling solutions.

Despite these limitations, Karpenter is still an excellent choice for many Kubernetes users. Its ability to optimize resource utilization, customize scaling behaviors, and reduce costs make it a compelling option for many organizations.

Conclusion

Karpenter is a powerful autoscaling solution that offers many advantages over the standard cluster autoscaler. With its ability to optimize resource utilization, customized scaling, and ease of use, Karpenter is a must-have tool for Kubernetes users. Follow the steps above to get started with Karpenter today!

Stay tuned for Part 2 of this blog, where we'll provide a step-by-step guide on how to install and configure Karpenter. Part 2 will be more technical, so if you're interested in getting started with Karpenter, be sure to check it out!

References

Follow me at LinkedIn and Twitter.