Forem: Nagesh Bansal

Discovering Chaos: My LFX Mentorship Journey with LitmusChaos

Nagesh Bansal — Fri, 01 Sep 2023 17:22:12 +0000

Diving into Kubernetes as a newcomer can be quite overwhelming. Have you ever thought about contributing to CNCF projects? If so, you've probably had a lot of questions. Where do you begin? Which project is best suited to you? And how can you properly engage the community in order to get answers?

I can surely relate to these concerns because I've experienced them myself. The CNCF arena is home to many projects, each excelling in its own field and requiring specific prerequisites.

As I navigated this complex landscape, I came across a valuable resource - 'LFX Mentorships for CNCF projects.' It was like discovering a powerful tool that turned me into a Kubernetes superhero, much like Thor."

"So, I'm here to share my journey and shed light on how I entered the world of chaos engineering, particularly with LitmusChaos."

What Is the LFX Mentorship Program and How Can You Apply?

The LFX Mentorship Program is a targeted 12-week initiative tailored to connect emerging contributors with seasoned mentors within the open-source realm. Its primary goal is to foster both personal and professional development by offering mentees invaluable guidance, unwavering support, and valuable insights.

To learn more about the program, be sure to check out the comprehensive LFX Mentorship Docs.

The application process for this program is uncomplicated. It starts with the creation of an LFX Mentee account, followed by the selection of projects from the platform. As part of your application, you'll need to submit both a resume and a cover letter. It's worth emphasizing that this isn't the final step, and this is where many students often make mistakes.

While applying for the project, I mentioned the approach I may use in the cover letter. I read the project Description and then created a github gist for the approach. So getting to know the project before applying is a good idea.

Choosing my Destiny: Project Pick and Confirmation

Ah, the moment of truth had arrived - it was decision time in my mentorship journey, and I had to pick a project. After setting up my LFX Mentee account and exploring the available projects, I had a list of options to consider. Each project seemed interesting in its own way.

But there was one project that stood out: "Enhance Chaos-Operator and Chaos-Exporter" by LitmusChaos. I decided to go for it, as I want to work with litmuschaos from 1st Term of LFX mentorship, and this project aligned my interest for Operator and Prometheus.

With a simple click on the "Apply" button, I threw my digital dice into the ring and submitted my application. Now, all I had to do was wait for a confirmation email.

The email arrived, and the subject line said it all, "Congratulations! You're In!" My excitement knew no bounds as I read the email, knowing that I had secured my place in the LitmusChaos for LFX mentorship program.

With my choice made and confirmation in hand, I was ready for the next chapter of my mentorship journey, where I'd explore the world of Chaos Engineering with LitmusChaos. The adventure was about to begin, and I couldn't wait to dive in.

Getting My Hands Dirty: The Work I Took On

With my project selected and my spot in the mentorship program secured, it was time to roll up my sleeves and dive into the thick of things. In this section, I'll walk you through the tasks, challenges, and discoveries I encountered as I got to work on the project. Let's jump right in!

My Project Included to work on several Litmuschaos Components, i.e chaos-exporter, chaos-operator, litmusctl.
and !!!
Shubham Chaudhary, Vansh Bhatia were my mentors during this mentorship. Later on, Saranya Jena, Sarthak Jain also joins.I had weekly sync-up meetings with them. In this meeting, we reviewed the goals we set in our previous meetings, discussed the progress, reviewed PRs I made, and identified any new goals or objectives that I would like to work towards. In our case, we documented our meetings in Docs, as shown in the photo below.

Firstly I started with reading Litmuschaos Docs, setting up my local environment, using the litmus and running the litmus on my local cluster. This helped my to get brief idea how litmus works, also Shubham bhaiya explained me several components of litmus and it's architecture.

Now, it was time to get into the project: I kicked off the project with the task of adding the Unit tests in Chaos-Exporter, with weekly sync meeting being held, and discussion , i raised my First PR. From this task I learned about unit testing in golang, mocking the function and a lot more things.

Now, Let's get to second task, so now Sarthak and Saranya joins us, and i've been assigned the task to make litmusctl, a command-line utility developed by the LitmusChaos community, offers a powerful set of commands to manage, configure, and control various aspects of chaos engineering experiments. We're keeping tracks of tasks in a docs.

As I was working on this task, I faced a lot of challenges but everything set sort out in the weekly sync up meetings. This task took me a long time, and finally raised the PR. Now it's was time test the changes, review the PR and after all resolving failing check again and again ( and force push ) we got something:

and amazing part is that, the changes introduced by were part of new release of litmusctl:

I've been contributing since the mentorship started, making small bugs fixed to introducing new changes. All of these contributions made the mentorship more interesting. If i started to explain all of these may this blog will never end. (just exaggerating)

Okay, you did this and that

This mentorship has been great, working on several projects with amazing mentors, will play a crucial role in my open-source journey.
Throughout my mentorship, I created a total of 5 pull requests. Out of these, 2 of them were successfully merged into the project, while 3 were still under review.
I also create a blog post 'Exploring litmusctl: A Comprehensive Guide' to Litmus Blogs. This blog explains how to use new release of Litmusctl.

LitmusChaos hosts Community and Contributor meeting every month and I had a chance to present my work and show it's demo in meet.

And i got graduated from LFX mentorship.. But exploring chaos doesn't end here.

Conclusion: From Chaos to Clarity

In summary, my journey through the LFX Mentorship Program with LitmusChaos has been both challenging and rewarding. Navigating the vast CNCF project landscape as a newcomer can be overwhelming, but with the right resources and guidance, it becomes an exciting adventure.

The LFX Mentorship Program connected me with experienced mentors who provided crucial support throughout my journey. Key takeaways include the importance of selecting a project aligned with your interests, submitting a well-prepared application, and actively engaging with mentors and the community.

During the mentorship, I made meaningful contributions to LitmusChaos. These contributions not only enhanced my skills but also enriched the LitmusChaos ecosystem and helped others.

As I graduate from the program, I look forward to continuing my exploration of Chaos Engineering and open source. To those considering a similar journey, I encourage you to embrace the opportunity, learn, and grow in the open-source world. Your adventure awaits!

And before we sign off, a big shoutout and heartfelt thanks go to the amazing folks —the Litmus team! Your dedication to crafting a tool as awesome as litmusc doesn't go unnoticed.

Discover more about LitmusChaos by becoming part of our community on GitHub and Slack.

It's time to let the chaos begin! 🚀🔥

Nagesh Bansal
LitmusChaos Contributor
Linkedin | Github

Exploring litmusctl: A Comprehensive Guide

Nagesh Bansal — Wed, 23 Aug 2023 13:36:41 +0000

This blog is a guide that explains how to run LitmusChaos on a remote cluster using Litmusctl. It's not just about scheduling Chaos Scenarios on your main cluster – Litmus also lets you set up Chaos on a remote cluster.

Litmusctl, a command-line utility developed by the LitmusChaos community, offers a powerful set of commands to manage, configure, and control various aspects of chaos engineering experiments.
In this guide, we will dive into the syntax, usage, and capabilities of litmusctl, making it easier for you to leverage this tool effectively.

Pre-Requisites

To run this experiment, we need a few things beforehand:

Minikube cluster. Recommended sizing of at least: 2 vCPUs, 8GB memory, 10GB of disk space (for persistent volume creation)
Helm3 or Kubectl
Access to Internet from Cluster
Have Litmus 3.0.0-beta9 or later installed.
A Google Cloud Account
A GCP project with privileges to access GKE.

Setting Up LitmusChaos Cluster on GCP

I'm using the GCP Kubernetes Engine to Run a LitmusChaos Chaoscenter instance on a remote cluster.
PS: You can use some other alternatives or maybe even a local cluster using minikube.

Install Litmus

Mongo installation via Helm - Bitnami Mongo

helm repo add bitnami https://charts.bitnami.com/bitnami

mongo-values.yml

auth:
  enabled: true
  rootPassword: "1234"
  # -- existingSecret Existing secret with MongoDB(&reg;) credentials (keys: `mongodb-passwords`, `mongodb-root-password`, `mongodb-metrics-password`, ` mongodb-replica-set-key`)
  existingSecret: ""
architecture: replicaset
replicaCount: 3
persistence:
  enabled: true
volumePermissions:
  enabled: true
metrics:
  enabled: false
  prometheusRule:
    enabled: false

# bitnami/mongodb is not yet supported on ARM.
# Using unofficial tools to build bitnami/mongodb (arm64 support)
# more info: https://github.com/ZCube/bitnami-compat
#image:
#  registry: ghcr.io/zcube
#  repository: bitnami-compat/mongodb
#  tag: 6.0.5

helm install my-release bitnami/mongodb --values mongo-values.yml -n litmus --create-namespace

Apply the Manifest

Applying the manifest file will install all the required service account configuration and ChaosCenter.

kubectl apply -f https://litmuschaos.github.io/litmus/3.0.0-beta10/litmus-3.0.0-beta10.yaml

Verify the Installation

Check if the Litmus control plane components comprising the Web-UI (frontend), GraphQL server(server), Auth-server & MongoDB pods are created and running successfully.

litmusctl-guide@cloudshell:~ (litmus-393109)$ kubectl get po -n litmus
NAME                                       READY   STATUS    RESTARTS     
litmusportal-auth-server-78dfb9f65-hq4lm   1/1     Running   0     
litmusportal-frontend-8cd79bb47-88qgm      1/1     Running   0
litmusportal-server-7b85867b95-j2ndz       1/1     Running   0         
my-release-mongodb-0                       1/1     Running   0    
my-release-mongodb-1                       1/1     Running   0
my-release-mongodb-2                       1/1     Running   0 
my-release-mongodb-arbiter-0               1/1     Running   0

It might take a few minutes for all the pods to have a running state. Give it some time.

Access the ChaosCenter

To access the litmus frontend, we need to know what service exposes this frontend. To get a list of running services in the litmus namespace, run the below command.

kubectl get svc -n litmus
NAME                                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                       
litmusportal-auth-server-service      NodePort    10.8.xxx.xxx   <none>        9003:xxxxx/TCP,3030:xxxxx/TCP
litmusportal-frontend-service         NodePort    10.8.xxx.xxx   <none>        9091:xxxxx/TCP               
litmusportal-server-service           NodePort    10.8.xxx.xxx   <none>      9002:xxxxx/TCP,8000:xxxxx/TCP
my-release-mongodb-arbiter-headless   ClusterIP   None           <none>        27017/TCP                    
my-release-mongodb-headless           ClusterIP   None           <none>        27017/TCP

From the above, we can see that the frontend is exposed via a NodePort service and it has the name of litmusportal-frontend-service. It exposes this service on a TCP port that looks something like 9091:xxxxx/TCP where xxxxx is the corresponding node port of the service.

To access the ChaosCenter, we need to expose this port by using a firewall rule. We can do this by running this command:

Expose the node port

gcloud compute firewall-rules create frontend-service-rule --allow tcp:<NODE_PORT>

Replace the NODE_PORT in the command above with your service's node port. You should expose the firewall for both the litmusportal-frontend-service and the litmusportal-server-service.

You can now access the ChaosCenter, but first, you need to get the external IPs of your node.

Get node external IPs

nageshbansal59@cloudshell:~ (litmus-393109)$ kubectl get nodes -o wide
NAME                                           STATUS   ROLES    AGE   VERSION            INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gk3-autopilot-cluster-1-pool-1-243efeac-8xwq   Ready    <none>   19d   v1.26.5-gke.1400   10.xxx.xx.xxx   34.xxx.xx.xxx   Container-Optimized OS from Google   5.15.107+        containerd://1.6.18
gk3-autopilot-cluster-1-pool-1-936f0a64-4rmh   Ready    <none>   19d   v1.26.5-gke.1400   10.xxx.xx.xxx   34.xxx.xx.xxx    Container-Optimized OS from Google   5.15.107+        containerd://1.6.18
gk3-autopilot-cluster-1-pool-1-dcd5ea45-xx28   Ready    <none>   8d    v1.26.5-gke.1400   10.xxx.xx.xxx   34.xxx.xx.xxx    Container-Optimized OS from Google   5.15.107+        containerd://1.6.18

Copy the external IP of any of the above nodes and paste it into your browser's URL, followed by a :xxxxx where xxxxx corresponds to your node port. You will be presented with a screen showing the ChaosCenter's homepage.

Accessing the Chaos-Center

Use Following Credinatls to login into the Chaos-Center:

Username: admin
Password: litmus

and here is the your Chaos-Center:

Litmusctl

The Litmuschaos command-line tool, litmusctl, allows you to manage litmuschaos's execution plane. You can use litmusctl to connect Chaos Infrastructures, create project, schedule Chaos Experiments, disconnect Chaos Experiments and manage multiple litmuschaos accounts.

Litmusctl binaries can be downloaded from the Litmusctl Releases according to the Compatibility Matrix.

As for now we're using the Litmus-3.0.0-beta10, so litmusctl-0.23.0 will be the version you should download according to OS.

NOTE: The binary should be downloaded to the target cluster and not the GKE cluster where the control plane is situated.

Installing litmusctl

For Linux/MacOS

Extract the binary

tar -zxvf litmusctl-<OS>-<ARCH>-<VERSION>.tar.gz

Provide necessary permissions

chmod +x litmusctl

Move the litmusctl binary to /usr/local/bin/litmusctl. Note: Make sure to use root user or use sudo as a prefix

mv litmusctl /usr/local/bin/litmusctl

Verify Installation

>> litmusctl version 
Litmusctl version:  0.23.0
Compatible ChaosCenter versions:

Using Litmusctl commands

litmusctl commands in Linux/macOS:

litmusctl <command> <subcommand> <subcommand> [options and parameters]

For using the v0.23.0 with Litmus-3.0.0-beta9 or later you can refer to the User Guide for v0.23.0

Connecting a Chaos Infrastucture to Chaos-Center

Litmusctl has the following requirements:

kubeconfig - litmusctl needs the kubeconfig of the k8s cluster where we need to connect litmus chaos delegates. The CLI currently uses the default path of kubeconfig.
kubectl - litmusctl is using kubectl under the hood to apply the manifest. To install kubectl, follow: kubectl

NOTE : I'm using a Minikube Cluster Locally.

To setup an account with litmusctl

litmusctl config set-account

Next, you need to enter ChaosCenter details to login into your ChaosCenter account. Fields to be filled in:

ChaosCenter URL: Enter the URL used to access the ChaosCenter.(The Url we used to get to the frontend)

Example, https://34.xxx.xx.xxx:XXXX

Username: Enter your ChaosCenter username.
Password: Enter your ChaosCenter password.

Host endpoint where litmus is installed: 34.xxx.xx.xxx
Username [Default: admin]: admin

Password:
account.username/admin configured

To Get a List of Projects

litmusctl get projects

To connect a Chaos Infrastructure in a cluster mode

Before a connecting a new Infrastructure check if you have an existing environment in your chaos-center.

If you don't have any, new environments can be created using the command:

litmusctl create chaos-environment      

Enter the Project ID: f20b7907-8644-42ad-b813-fbac39fb8eb4

Enter the Environment Name: demo-env

🚀 New Chaos Environment creation successful!! 🎉
EnvironmentID: demo_env

Now finally connecting a new Infrastructure

litmusctl connect chaos-infra

After Running this command, several details are required to connect the new Infrastructure:

And, after applying the manifest to your local cluster, we will get:

also, you can see the new Infrastructure in chaos-center:

Now,I would suggest you to try to run a Chaos Experiment Using the Litmusctl and Chaos-Center both and, let me know how it goes. ( Refer to Usage Guide if you hit any blocker :-).

And before we sign off, a big shoutout and heartfelt thanks go to the amazing folks behind the scenes—the Litmus team! Your dedication to crafting a tool as awesome as litmusctl doesn't go unnoticed

Discover more about LitmusChaos by becoming part of our community on GitHub and Slack.

It's time to let the chaos begin! 🚀🔥

Nagesh Bansal
LitmusChaos Contributor
Linkedin | Github