Forem: LABOUARDY Mohamed

Deploy Komiser to AWS with Terraform

LABOUARDY Mohamed — Mon, 24 Apr 2023 18:57:57 +0000

Komiser is an open-source cloud-agnostic inventory solution that gives you a holistic view of your cloud resources and helps you uncover insights about your infrastructure such as cost wasted, security threats, and compliance issues.

This tutorial will cover all the necessary steps for deploying Komiser on AWS using Terraform. By the end of this tutorial, we’ll deploy the below architecture with the following components:

Provision an EC2 instance to host the Komiser container, and ensure that inbound and outbound traffic to the instance is restricted using a security group.
Deploy an ELB in front of the EC2 instance to manage traffic distribution, and configure a security group for the ELB.
Create an IAM instance profile and attach it to the EC2 instance, including the Komiser recommended IAM policy to manage access to AWS resources.
Set up an alias record in Route 53 to direct traffic to the ELB DNS with a user-friendly URL.

All Terraform templates used in this tutorial can be found in the GitHub repository.

To get started, define your backend and declare AWS as your provider in the terraform.tf file. In this example, S3 is used as the backend for storing Terraform state files. Once done, run terraform init to download the AWS module.

Next, declare an EC2 instance in ec2.tf file with aws_instance resource. The resource uses Amazon Linux 2 as an AMI which is obtained using the data block and the aws_ami data source. The instance type is t2.medium (recommended size to host Komiser) and uses a public IP address and a security group that allows traffic on port 22 for SSH access and 3000 for serving the Komiser dashboard. It also attaches an IAM instance profile with the permissions required by Komiser to build your asset inventory.

The provisioned blocks define a series of file transfers and commands to execute on the EC2 instance after it’s launched. The first three provisioner blocks upload files from the local machine to the EC2 instance. The last provisioner block executes remote commands on the EC2 instance, which installs some needed dependencies by running a bash script that is transferred to one of the previous provisioner blocks and deploys Komiser as a Docker container.

It’s important to note that when using Komiser in a production environment, certain additional improvements should be taken to ensure security and scalability.

Firstly, it’s recommended to deploy the Komiser instance within a private subnet and restrict SSH access only from a trusted CIDR block or a bastion host. This helps to prevent unauthorized access to the instance and reduces the risk of security threats.

Secondly, automation tools such as Ansible or Packer can be leveraged to further optimize the deployment process. Using Packer, for example, allows for creating a custom AMI that includes all the necessary software and configurations for running Komiser.

The install.sh script installs Docker Community Edition (CE) and Docker Compose. It also adds the ec2-user to the docker group, allowing us to run Docker commands without needing to use sudo.

Once those tools are installed, the deployment of the Komiser container can be initiated by executing the command docker-compose up with reference to the docker-compose.yml file. This will result in the deployment of the latest version of the Komiser image, which is at the time of writing this post v3.0.11.

The container is configured to use a config.toml file that connects to the running AWS account and stores data in an SQLite database. Additionally, the container serves the Komiser dashboard through port 3000.

In the iam.tf file, the next step is to define AWS IAM resources that enable the Komiser EC2 instance to assume an IAM role with the appropriate permissions attached to it.

Next, in elb.tf define a load balancer resource that forwards traffic into the EC2 instance running the Komiser container on port 3000. The ELB is configured with two listeners: one for HTTPS traffic and the other for HTTP traffic. The HTTPS listener is configured with an SSL certificate (requested through ACM) that is specified as a variable.

The health_check block specifies a health check configuration for the ELB. It specifies the target to check for health as “TCP:3000”. The health check is configured to check the instances every 5 seconds, with a timeout of 3 seconds, and a threshold of 2 checks for both healthy and unhealthy responses.

Finally, in route53.tf creates a new AWS Route 53 record for a domain name that points to the ELB resource using an alias record. The record type specified is “A” for an IPv4 address.

This creates a Route 53 record that maps the domain name to the ELB, making it possible for users to access Komiser running on the ELB through a user-friendly URL.

After defining variables and outputs, running terraform plan will generate an execution plan detailing the changes that will be made to the infrastructure. Running terraform apply will apply these changes, resulting in the deployment of the 9 new resources necessary for running Komiser on AWS.

After the resources have been successfully provisioned, you can easily access the Komiser dashboard by navigating to https://demo.domain.com. Once accessed, you will be presented with a comprehensive breakdown of your AWS resources, including their associated costs.

Congrats! You’ve successfully deployed Komiser with Terraform.

You can now leverage Komiser’s holistic view to take control of your cloud usage and optimize your resources for maximum efficiency and cost savings.

Managing Multiple Environments with Terraform

LABOUARDY Mohamed — Sat, 15 Apr 2023 13:22:04 +0000

Get ready for another exciting edition of the DevOps weekly newsletter! This week, I’ve got a lot in store for you.

Discover how to improve AWS dead-letter queues, automate multiple AWS environments with Terraform, and explore how Amazon practices continuous deployment. We’ll also explore the dangers of relying on ChatGPT for security guidance and how database sharding works in MySQL and Postgres.

Plus, discover what happens when you leak AWS credentials and explore the ultimate guide to multi-tenancy in Kubernetes. And don’t miss a funny video of the week, highlighting the main reasons people hate Amazon Web Services.

As always, I’ve got some fantastic open-source projects for you to check out, including Tart, a virtualization toolset for building, running, and managing macOS and Linux virtual machines on Apple Silicon, Chroma, an open-source embedding database, and a curated list of awesome PostgreSQL software, libraries, tools, and resources inspired by awesome-mysql. We’ll also look at Timoni, a package manager for Kubernetes powered by CUE and inspired by Helm.

This is one DevOps newsletter you won’t want to miss!

Tutorials of the week

👀 “Improving our dead-letter queues” — This blog post addresses a previous incident where messages placed on a AWS dead-letter queue were lost, and how a better process was implemented — Read more »

🚀 “Automating multiple environments with Terraform” — How to manage a central main account for shared infrastructure with a dev, test, and prod account — Read more »

🔌 “Automating safe, hands-off deployments” — Strategies for continuously deploying to production while balancing safety and speed — Read more »

🔒 “Please don’t use GPT for Security guidance” — This is a really bad idea and this is why — Read more »

⭐️ “How does database sharding work?” — How database sharding works on MySQL & Postgres, how to think about implementing your own sharded database, and some useful tools out there that can help — Read more »

🔥 “Uptime guarantees — a pragmatic perspective” — Engineering for 99.5% uptime is more cost-effective than 99.99% for most startups — Read more »

😵 “What happens when you leak AWS credentials” — and how AWS minimizes the damage — Read more »

Open-source projects of the week

1️⃣ Tart is a virtualization toolset to build, run and manage macOS and Linux virtual machines (VMs) on Apple Silicon — Learn more »

2️⃣ Chroma is an open-source embedding database. Fastest way to build Python or JavaScript LLM apps with memory — Learn more »

3️⃣ NoiSQL shows how to play sound and music with declarative SQL queries — Learn more »

4️⃣ Timoni is a package manager for Kubernetes, powered by CUE and inspired by Helm — Learn more »

5️⃣ Tabby is a self-hosted AI coding assistant. An open source/on-prem alternative to GitHub Copilot — Learn more »

6️⃣ A curated list of awesome PostgreSQL software, libraries, tools and resources, inspired by awesome-mysql — Learn more »

Thread of the week

Full thread here.

Memes of the week

I hope you enjoy this week’s newsletter! Share it with a friend or colleague if you find it helpful, drop me an email or send me a DM on Twitter about topics you’d like to hear about in future editions.

Build a Serverless Gym App with ChatGPT, Twilio and WhatsApp

LABOUARDY Mohamed — Mon, 03 Apr 2023 17:05:42 +0000

We have recently started hosting virtual workshops in our Discord community called “Wardens Assembly”. These monthly events cover a variety of tech topics. The first event was about building a Serverless gym app that sends a workout plan to your WhatsApp number using ChatGPT. In case you missed the event, you can watch it completely here:

In case you want to cut straight to the chase, this tutorial is for you as it covers only the key parts of building a Serverless app, including:

Building a Serverless app using Golang & AWS Lambda
Scheduling Lambda with cron expression triggers
Integrating ChatGPT 4 with AWS Lambda
Sending WhatsApp messages via the Twilio SDK
Streamlining deployment with GitHub Actions & AWS SAM

Before jumping into the code, the diagram below summarizes the architecture we’re going to build by end of this tutorial:

The goal is to create a Lambda function in Go, which will communicate with ChatGPT. This function will send a prompt to ChatGPT API and then use Twilio to send a workout plan to our WhatsApp number at a specific schedule determined by an EventBridge Rule. Moreover, we will leverage AWS SAM and GitHub Actions to automate the infrastructure build and deployment, as well as the CI/CD pipeline.

You can find all the source code used in this tutorial on GitHub.

Building Lambda Function

To get started, from your terminal create a main.go file, initialize a go project, and install the AWS Lambda package with the following commands:

go mod init workout-generator go get github.com/aws/aws-lambda-go

Next, declare the handler function in main.go. The main function calls the lambda handler by calling the lambda.Start method_._

Next, to integrate with OpenAI, you will need to download the OpenAI Go wrapper library:

go get github.com/sashabaranov/go-openai

Then, update the handler, and create an OpenAI client by passing your OpenAI token. Next, start a chat using the CreateChatCompletion method and pass a prompt, and setGPT3Dot5Turbo as the target model (which is the underlying name for ChatGPT). The library also supports other models, such as ChatGPT, GPT-4, DALL·E 2, and Whisper.

The OPENAI_TOKEN value can be generated from the OpenAI platform. Make sure to save the key in a safe place, as it will only be shown to you once.

To send the workout plan to our WhatsApp number, we need to integrate Twilio. To do so, you will need to sign up for a Twilio account, sign in to your existing account, and activate the Twilio Sandbox for WhatsApp. Follow the steps below:

Sign up for a free Twilio account.
Activate the Twilio Sandbox for WhatsApp.
Select a number from the available sandbox numbers to activate your sandbox.
Send join to your Sandbox number in WhatsApp to join your Sandbox.

After sending the message, Twilio should reply with a confirmation message, as shown in the screenshot below:

Now to integrate with our app, install the Twilio Go package with the following command:

go get https://github.com/twilio/twilio-go

Next, add the following code snippet to the Lambda handler. It creates a Twilio client by passing the credentials as environment variables. The Account SID and Auth Token can be found here and should be set as the values for the environment variables TWILIO_USERNAME and TWILIO_PASSWORD respectively. Finally, it uses the CreateMessage method to send the workout plan generated by ChatGPT.

That’s it. Our function handler is ready to be deployed to AWS Lambda!

Deploying Serverless Stack with SAM

For the deployment part, we’re going to use AWS Serverless Application Model (SAM). Once you’ve installed the SAM CLI, create a template.yml that declares a Lambda function called “WorkoutGenerator”, including its source code location, handler function, runtime environment, memory size, and timeout duration.

The template also defines a set of environment variables for the function. These variables are resolved using the AWS Systems Manager Parameter Store, which is a service that stores secure strings and parameters. The variables specified in this template include the OpenAI token, Twilio account credentials, and phone numbers for sending and receiving WhatsApp messages.

Next, use the AWS SAM CLI to build the application and prepare for deployment by running the sam build command. Finally, run the sam deploy — guided command to deploy the AWS resources by provisioning an AWS CloudFormation stack:

The Lambda function is now deployed and running in the AWS Cloud! You can test it out, by triggering the Lambda function manually from the AWS Console. A WhatsApp message should be received with a workout plan generated by ChatGPT as follows:

However, our goal is to have our workout plan generated automatically, ideally before we begin our gym workout. To achieve this, we need to trigger our Lambda function at a specific schedule. We can use an EventBridge rule to define a cron expression that invokes our function every weekday at 7 PM. Update the SAM template below and run it again:

If you head back to the Lambda dashboard, you should see an EventBridge Rule trigger, as shown in the screenshot below:

With our application being completed, let’s build a CI/CD pipeline to automate the deployment process through GitHub Actions.

Defining a CI/CD Pipeline with GitHub Actions

Once your source code is pushed to a remote GitHub repository, create a release.yml file under the .github/workflows folder with the following steps:

Check out the repository under $GITHUB_WORKSPACE, so the workflow can access it.
Set up Python environment and install SAM CLI.
Configure AWS credentials from secrets. Make sure to add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as a secret under the GitHub repository.
Run the sam build and sam deploy commands in a non-interactive mode.

Once the pipeline is defined, push the changes to the remote repository.

We can test out the pipeline by improving the ChatGPT prompt to generate a workout plan with only weightlifting exercises. Push the changes, the pipeline will be triggered and new changes will be deployed to AWS as shown below:

To improve the workout plan, provide your age, gender, weight, height, and one rep max in the ChatGPT prompt.

Congratulations! You have successfully built a personal gym workout generator. We would love to see your implementation. Create a similar Serverless app that integrates with ChatGPT and share it on Twitter to enter the competition. Tag Tailwarden on Twitter and add the hashtag #WardensAssemblyChallenge. The three winners will be chosen on Friday, April 28th.

Kubernetes Broke Reddit

LABOUARDY Mohamed — Fri, 31 Mar 2023 15:58:19 +0000

Get ready to supercharge your DevOps knowledge with another jam-packed edition of our weekly DevOps newsletter! Unravel the mystery behind unexpected charges on your AWS bill, get practical tips for rightsizing your Kubernetes workloads, and explore the benchmarking results of the AWS SDK v2 and v3.

Dive into using Terraform to deploy a Counter-Strike server, learn about the notorious Lambda — S3 infinite loop, and discover how to implement magic links with Amazon Cognito. Plus, we’ll discuss testing AWS Serverless Microservices and share an insightful Reddit incident postmortem.

As always, our open-source projects of the week won’t disappoint! Get your hands on a bash script to build a minimal Linux operating system just to play Doom. An open-source GitHub Copilot server, the cloud-nuke tool for cleaning up your cloud accounts, and others.

Lastly, don’t miss my video tutorial on building a Serverless gym app that sends personalized workout plans to your WhatsApp using ChatGPT, Twilio, AWS SAM, and GitHub Actions from scratch, along with other exciting content!

Tutorials of the week

💰 “Unexpected charges on your AWS bill” — Are you a new AWS user who has experienced bill shock while being in a free-tier plan or a cloud practitioner struggling to understand your team’s cloud expenses? This post can be useful — Read more »

🔨 “Practical tips for rightsizing your Kubernetes workloads” — How resources are allocated in K8s envs and tips for rightsizing your workloads for cost efficiency and performance — Read more »

📊 “Benchmarking the AWS SDK” — I would have expected the v3 to perform better than v2 — Read more »

🎮 “Using Terraform to deploy a Counter-Strike” — For the geeks out there, learn how to build CS: GO server with Terraform from scratch — Read more »

😅 “The S3 — Lambda death spiral loop” — The infamous Lambda — S3 infinite loop and easiest way to lose money — Read more »

🔒 “Implementing magic links with Amazon Cognito” — Tutorial on how to build a Lambda function that sends an email with a time-limited URL for passwordless authentication — Read more »

🚀 “Managing risk as an SRE” — While evangelism can be a cap on advancement as an SRE, risk management ability is a hard requirement — Read more »

🧪 “How to test AWS Serverless Microservices” — It covers unit, solitary, sociable, integration, API, and E2E tests — Read more »

Open source projects of the week

1️⃣ A bash script to build a minimal Linux operating system just to play Doom — Learn more »

2️⃣ Autometrics is a set of open-source libraries that make it fun and easy to understand the performance of your code in production.- Learn more »

3️⃣ FauxPilot is an open-source GitHub Copilot server — Learn more »

4️⃣ cloud-nuke is a tool for cleaning up your cloud accounts by nuking (deleting) all resources within it — Learn more »

5️⃣ Markprompt is an open-source GPT-4 platform for Markdown, Markdoc and MDX with built-in analytics — Learn more »

6️⃣ Untitled Goose Tool is an incident response tool that runs a full investigation against a customer’s AzureAD, Azure, and M365 environments — Learn more »

Thread of the week

Outage due to upgrading to a newer Kubernetes version…

Memes of the week

Unexpected charges on your AWS bill

LABOUARDY Mohamed — Mon, 27 Mar 2023 13:36:16 +0000

Are you a new AWS user who has experienced bill shock while being in a free-tier plan or a professional cloud practitioner struggling to understand your team’s cloud expenses? If so, you’re not alone. Many AWS users are surprised by unexpected charges on their monthly bills, which can significantly increase the overall cost of using AWS services. As companies’ cloud environments become more complex, lack of visibility can lead to unpredictable cloud bills and budget overruns. In this blog post, we will cover the most common unexpected charges on your AWS bill and provide tips on how to avoid them, so you can optimize your cloud costs and avoid unpleasant surprises.

It’s worth mentioning that the list is not exhaustive and covers only the most common ways in which money is wasted on AWS.

Paying for unused resources

One of the main common unexpected charges is related to idle resources. AWS charges for resources that are not actively used, such as idle EC2 instances, RDS databases, and Elastic Load Balancers. These charges can accumulate over time and result in significant bills. To avoid this, you can use AWS Auto Scaling to automatically adjust the number of resources based on demand. You can also use AWS Reserved Instances or Savings Plans to reduce costs by committing to a one-year or three-year term or use Spot Instances for non-critical jobs or workloads.

Another common source of idle resources is unattached EBS volumes and unused Elastic IPs. To avoid these charges appearing on your bill, create a policy that automatically deletes any unused EBS volumes or EIPs.

Infrastructure Drift

One major cause of infrastructure drift is the creation of resources outside of the established IaC tools such as Terraform, CloudFormation, and Pulumi, or without proper approval. When this happens, the infrastructure state is not adequately described or persisted, and the changes made to the infrastructure go unnoticed (aka shadow IT activity).

Until you have total visibility across your environment and have implemented measures to prevent the use of cloud consoles, infrastructure drift is likely to contribute to your AWS bill.

Data requests, transfers, and retrievals

Another unexpected item that can pop up on your AWS bill is related to data transfer. AWS charges for data transfer within the platform, as well as data transfer to and from the internet. Many users are unaware of this charge and end up with significant increases. To avoid this, you can use AWS services in the same region or availability zone, which is usually free of charge. You can also use AWS Direct Connect for data transfer between your data center and AWS, which can significantly reduce data transfer charges.

CloudWatch Logs

CloudWatch is the primary source of truth for monitoring the overall health and storing logs of active AWS cloud services. However, it is also notorious for surprise bill spikes due to the complexity of its pricing model.

The pricing is determined by various factors, such as the number of custom metrics, alarms, and dashboards, logs ingested, stored, and analyzed, and the use of contributor insights rules and synthetics canary runs.

The most common way of rapidly driving up costs is by leaving the default retention period. This is especially true for AWS Lambda, which creates an automatic log group with an indefinite retention setting. It’s also important to use alarms and dashboards for key metrics only that way avoiding unnecessary alerts and visualizations.

AWS Free Tier expired or usage exceeds

AWS is pretty generous with free-tier plans but without proper monitoring, you can exceed the free usage limits in a breeze. The good news is you can monitor usage through the AWS Management Console and track free tier usage. Any usage beyond the free tier limit or after a free trial has ended is charged at standard rates. To avoid charges, set up alerts to notify you before the free tier expires or usage exceeds the limit.

AWS Free Tier usage alerts automatically notify you over email when you exceed 85 percent of your Free Tier limit for each service.

Underutilized Reserved Instances and Savings Plans

When it comes to AWS EC2 costs, there are several recommendations that you can use to save money. One popular approach is to purchase Reserved Instances and Savings Plans. By doing so, companies can potentially save a lot of money on their monthly bills. However, it’s important to note that simply purchasing these plans isn’t enough. In order to fully reap the benefits of Reserved Instances and Savings Plans, you need to make sure that they are being used, monitored, and optimized effectively. Failure to do so can result in unexpected charges and higher costs.

Dynamic Environments

Elastic Beanstalk is designed to ensure that all necessary resources are running. As a result, it will automatically relaunch any services that you stop. To prevent this, you must terminate your Elastic Beanstalk environment before terminating any resources that Elastic Beanstalk has created.

Similarly, auto-scaling groups are designed to maintain a minimum number of EC2 instances running. Ensure that you terminate your ASG or update the scaling policies to avoid unexpected charges.

Preventing AWS Bill Shock

AWS has many services to help monitor billing. Setting an account-wide budget alert is a relatively easy first line of defense. Secondly, regularly review and tag your resources to identify any unused or idle resources that you can terminate or downsize to reduce costs. Thirdly, use AWS Cost Explorer to analyze and visualize your costs, identify cost trends, and optimize your spending. Finally, take advantage of AWS tools such as CloudWatch and AWS Config to monitor and optimize your resources continuously.

If you’re looking for an all-in-one platform, you can also leverage open-source tools like Komiser to build your cloud asset inventory, tag resources, set up budget alerts, and bring accountability to cloud spend.

Conclusion

Unexpected charges on AWS bills can put businesses out of the market. However, with the right practices and tools in place, companies can detect and troubleshoot overspending issues before they ever occur.

Whether you’re just starting out with AWS or you’re a seasoned DevOps engineer, Komiser can help you catch potential cost optimization opportunities early, before they become a larger problem.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

ChatGPT for DevOps

LABOUARDY Mohamed — Sun, 26 Mar 2023 17:02:05 +0000

Get ready for an exciting edition of our weekly DevOps newsletter! This week, we’ve got a diverse range of topics that will help you level up your skills and stay updated with the latest trends. Learn how to build an efficient pull request review process, implement passwordless authentication with Amazon Cognito, and build bedtime stories for children with AWS serverless services, ChatGPT, and DALL-E.

Plus, discover how to reduce your AWS billing cost from $7000+ to under $2000 dollar and run virtual machines like a Pod and orchestrate them via Kubernetes.

Don’t miss out on the open-source projects of the week, a podcast episode with cost-optimization tips for running containers on ECS, and my free workshop event on how to build a Serverless app with AWS Lambda, ChatGPT, and Twilio. Buckle up and let’s dive in!

Tutorials of the week

👀 “Building a scalable PR review process” — A great read on how to optimize the code review process for fast-growing organizations or teams — Read more »

🔒 “Passwordless authentication made easy with Cognito” — A hands-on tutorial on how to implement passwordless authentication with Amazon Cognito — Read more »

😅 “Implementing a Serverless story generation application with ChatGPT and DALL-E” — This post covers how to build a bedtime story for children with AWS serverless services, ChatGPT and DALL-E — Read more »

💰 “How I reduced our AWS billing cost from $7000+ to under $2000 dollar?” — Optimizing MySQL queries and proper indexing, API Gateway caching, CloudWatch logs retention, CloudFront caching, and other tips — Read more »

📦 “KubeVirt Part 1 — Run VMs like a Pod” — Learn how to run virtual machines like a Pod and orchestrate them via Kubernetes — Read more »

🤖 “Using ChatGPT for DevOps” — Tips and tricks to get the most out of using ChatGPT for DevOps — Read more »

🚀 “Inside logical replication in PostgreSQL” — This blog post will go through the fundamentals of Logical Replication and some use cases — Read more »

🔨 “How to own your own Docker Registry address” — Dodge the next Dockerpocalypse and setup your own Docker registry — Read more »

🔎 “Analyzing multi-gigabyte JSON files locally” — For data engineers, this tutorial can be helpful on how to leverage jq, Jupyter and Dask for large JSON files analysis — Read more »

Open-source projects of the week

1️⃣ SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence — Learn more »

2️⃣ A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system — Learn more »

3️⃣ Chainloop is an open source software supply chain control plane, a single source of truth for artifacts plus a declarative attestation process — Learn more »

4️⃣ Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON — Learn more »

5️⃣ Terraform Live Graph Extension for vscode is a plugin that allows you to generate a live Terraform graph as you code — Learn more »

6️⃣ libgsqlite is a SQLite extension that loads a Google Sheet as a virtual table — Learn more »

Tweet of the week

Lambda functions are cost-efficient they say 😅

Full thread

Memes of the week

I hope this summary has been helpful. Remember to subscribe to the newsletter to receive the latest DevOps trends in your inbox every week.

Postgres Explained

LABOUARDY Mohamed — Mon, 20 Mar 2023 05:34:42 +0000

Get ready for another action-packed edition of our weekly DevOps newsletter! This week, I’m covering a diverse range of topics that will help you level up your skills and stay up to date with the latest trends in DevOps. From exploring the ins and outs of Postgres architecture to learning how to troubleshoot common Kubernetes errors, I’ve got you covered.

Plus, we’ll be diving into the world of AWS with articles on Just-in-Time Access and Serverless AWS CDK pipeline best practices & patterns. We’ll also be discussing the importance of SREs being evangelists to be successful, and how to deal with devs pushing bad code to production.

And that’s not all — I’m sharing open-source projects of the week, including a DevOps framework based on getting things done, APE, Troubleshoot, and Helicone. So, whether you’re looking to automate your infrastructure with Terraform and Buildkite or learn how Discord stores trillions of messages, this week’s newsletter has got you covered. Don’t miss out — buckle up and let’s dive in!

Posts of the week

🔒 “The unreasonable effectiveness of just-in-time access” — If an attacker obtained one of your developer’s credentials, what access would they have? By adding a temporal dimension to access policies, the attack surface can be significantly reduced for many security-breach scenarios. That’s where just-in-time access comes in — Read more »

⭐️ “Demystifying the For vs Owns vs Watches controllers” — Theoretical side of these Kubernetes controller builders as well as their practices with a real-life examples — Read more »

🤖 “Automate your infrastructure with Terraform and Buildkite” — Learn how to use Buildkite to deploy your Terraform code changes — Read more »

🌎 “SRE evangelist” — SREs must be evangelists to be successful, making reliability more interesting and externalizing the feeling — Read more »

🔨 “How to identify and troubleshoot common Kubernetes errors” — Monitoring Kubernetes series that explains everything you need to quickly set up your Kubernetes clusters and monitor them — Read more »

🌥 “Inside Uber’s move to the cloud” — Uber has operated its own data centers for 9 years. What challenges did the company face, and why is it considering moving to the Cloud? — Read more »

✨ “Serverless AWS CDK pipeline best practices & patterns” — An opinionated discussion around how to set up, structure, and deploy your AWS CDK Serverless apps using CDK Pipelines in line with AWS best practice — Read more »

Projects of the week

1️⃣ Dozzle is a web-based app to monitor Docker logs. It doesn’t store any log files. It is for live monitoring of your container logs only — Learn more »

2️⃣ The Do Framework is a DevOps framework focused on simplicity, intuitiveness, and productivity. It helps you get more done — Learn more »

3️⃣ APE takes all of your AWS IAM policies and presents you with a single policy, summarizing all of their actual permissions — Learn more »

4️⃣ Troubleshoot is a kubectl plugin providing diagnostic tools for Kubernetes applications — Learn more »

5️⃣ Meshery is the cloud-native management plane offering lifecycle, configuration, and performance management of Kubernetes, service meshes, and your workloads. — Learn more »

6️⃣ Helicone is an open-source observability platform for GPT-3 users — Learn more »

Question of the week

Easy! revoke their access 💀

Full thread

Meme of the week

I hope this summary has been helpful. Remember to subscribe to the newsletter to receive the latest DevOps trends in your inbox every week.

Why Use Message Brokers

LABOUARDY Mohamed — Tue, 14 Mar 2023 09:10:11 +0000

Get ready for another jam-packed edition of our weekly DevOps newsletter! This week, I’m covering everything from monitoring production systems and using message brokers to explaining CDN in simple words and testing AWS Serverless & Lambda. Plus, I’ll be sharing tips for writing Terraform for unsupported resources, discussing AI-generated infrastructure-as-code, and highlighting the best book for hiring talented tech engineers.

Don’t forget to check out a podcast featuring IAM best practices, and our open-source projects of the week, including a web-based database interface, a tool for deploying apps with zero downtime, and an AWS Spot instances estimator. And that’s just the start — there’s plenty more to explore. So buckle up and let’s dive in!

Posts of the week

🔎 “Listing all AWS resources in an AWS account” — I recently wrote a blog post on how to list all your AWS resources and build your asset inventory to answer questions about your AWS infrastructure — Read more »

🧪 “Test In production: the ideal monitoring” — A few inputs on monitoring your production system for any regression bug that can be introduced while everyone constantly makes changes — Read more »

🗳 “Why use message brokers?” — Reduce pressure off downstream consumers, prevent messages/data from being lost, parallel processing, and others — Read more »

🌎 “Content Delivery Network (CDN): explained in simple words” — This post does a great job on explaining what a CDN is and how it works internally — Read more »

⭐️ “Guide to AWS Serverless & Lambda testing” — A practical guidelines for testing Serverless based apps, from mocking events to E2E tests — Read more »

✨ “Writing Terraform for unsupported resources” — TerraCurl is a utility Terraform provider that allows you to make managed and unmanaged API calls in their Terraform code — Read more »

🔥 “Move past incident response to reliability” — Remember when optimism and crossed fingers were our first line of incident response? @lethain outlines a better way — Read more »

🚀 “Ensuring smooth migration to Serverless” — Should you do performance testing if AWS says that a particular service has certain Service Level Objectives? If yes, what process should you follow? — Read more »

Projects of the week

1️⃣ Mathesar is an open source tool that provides a spreadsheet-like interface to a PostgreSQL database — Learn more »

2️⃣ A native desktop application that allows you to estimate the cost savings you can achieve in your AWS account by converting your AutoScaling Groups to Spot instances — Learn more »

3️⃣ MRSK deploys web apps anywhere from bare metal to cloud VMs using Docker with zero downtime — Learn more »

4️⃣ A verification engine on Kubernetes that enables verification of artifact security metadata and admits for deployment only those that comply with policies you create — Learn more »

5️⃣ A lightweight utility to dump AWS Fargate’s ECS containers environment variables locally — Learn more »

6️⃣ 30 days of Python programming challenge is a step-by-step guide to learning the Python programming language in 30 days — Learn more »

Question of the week

Development containers are a thing!

See thread

Meme of the week

I hope this summary has been helpful. Remember to subscribe to the newsletter to receive the latest DevOps trends in your inbox every week 🔥

How to find all resources in an AWS account

LABOUARDY Mohamed — Fri, 03 Mar 2023 17:02:01 +0000

When managing your cloud infrastructure on AWS, it’s important to have a comprehensive understanding of all the resources running in your AWS accounts. It’s crucial to be able to have reliable data and clear insight into areas such as:

Identifying resources that are idle, unmonitored, or exposed to security threats.
Understanding the cost breakdown and coverage of tags.
Keeping up-to-date and accurate audit information.
Evaluating whether your resources conform to specific governance controls.
Detecting any infrastructure drift and changes in configurations.

Without answers to those questions, you open the doors to cost wastage, security threats, and compliance issues.

This blog post provides guidance on the different tools available that can help you in locating and identifying all resources within your AWS account.

AWS Native Services

AWS provides several tools to help you identify and track resources in your account. Each tool has its own pros and cons, as we will see. The key distinction for cloud resource tracking is the scope of the tool and which resources are in its zone. Let’s dive in.

AWS Management Console

For anyone working with AWS, the AWS Management Console is a good place to begin. It provides access to a wide range of services and features. However, the console’s UI can be challenging to navigate, and it can be overwhelming. Additionally, you should have prior knowledge of the resources you are searching for and their location in the region. Otherwise, you may end up spending several hours browsing through multiple tabs and levels of hierarchy in the AWS Console to find answers to questions such as “What is the number of EC2 instances operating in our Frankfurt region?”

AWS Resource Groups

AWS Resource Groups is a better alternative to the AWS console. This service enables you to create a custom group of your resources, based on specific criteria such as tags or the resources in an AWS CloudFormation stack. By organizing and consolidating information in this way, you can easily track the resources used by individuals or application teams.

It’s worth noting that the service doesn’t support all AWS services (AWS services that work with AWS Resource Groups), as it was not specifically designed for resource discovery. Rather, the service is intended to group resources together based on predetermined tags or CF stack. Therefore, it may not be the best option if you’re looking for a tool to build your asset inventory.

AWS Config

AWS Config is a full-fledged asset inventory. It discovers all your running AWS resources and their configuration history as well as the resource relationships (e.g: find out if an EBS volume is attached to an EC2 instance associated with a security group).

The service also provides a rules engine that you can use to evaluate the configuration of resources against pre-defined rules or compliance policies. E.g: you can use SQL queries to find resources that are non-compliant AWS resources and export the results to JSON or CSV format for further benchmarks (e.g: CIS AWS Benchmarks, AWS Foundational Security Best Practices, or PCI DSS)

Despite those features, the AWS Config service does come with certain drawbacks, including:

As of the time of writing, AWS Config does not cover all types of resources (A list of supported services can be found here).
The more configuration items generated, the more expensive the service can become (See pricing).
AWS Config is best suited for AWS resources. Therefore, users operating in multi-cloud environments and organizations seeking configuration visibility for SaaS assets may require additional tools.
The service is not enabled by default, so users need to set it up in all regions for all their AWS accounts. For those with a considerable number of AWS accounts, this can result in significant effort.

AWS Cost Explorer and CloudWatch

It is also a good idea to take a look at Cost explorer once in a while and check whether we are charging our account unnecessarily. Billing information cannot provide a complete picture. But you can use the AWS Cost Explorer to slice your AWS cost by both AWS services, regions, and tags (if enabled). This can give you a starting point of where to further explore manually with AWS Config or Resource Explorer.

You can also leverage AWS CloudWatch to identify which resources are generating metrics so no resource goes untracked.

AWS Resource Explorer

AWS Resource Explorer is a service released last year that allows you to explore and discover the resources in your AWS account. It allows you to view, search, and filter the resources across all regions and services in your AWS account. The service is free of charge, making it a great alternative to other resource discovery mechanisms, such as AWS Config.

Resource Explorer was built with cross-region support from the very beginning. However, the list of resource types that can be discovered with Resource Explorer is quite short and does not support searching across multiple accounts inside an organization (It only works on an AWS account scope).

As such, you may want to consider alternative options that are more user-friendly and offer a more intuitive way to manage your resources on AWS.

Komiser

Komiser is an open-source cloud-agnostic asset inventory. It integrates with multiple cloud providers, builds a cloud asset inventory, and helps you break down your cost at the resource level.

Komiser comes with a resource inventory feature where you can have an active resource inventory of all your cloud resources along with relevant information such as source account, region, cost, and the tags that are applied to it. You can analyze cloud resource utilization and costs based on specific criteria, such as teams, applications, or cost centers. This approach enables the creation of custom views for engineering, finance, and product teams and promotes accountability for cloud expenses.

As you’re moving to a multi-cloud model, you would need a single place where you can manage all your cloud resources. By integrating with several cloud service providers (currently supporting AWS, Azure, Oracle, DigitalOcean, Civo, Tencent Linode, Kubernetes, and Scaleway), Komiser can swiftly generate your cloud asset inventory. This allows you to utilize its powerful filter system to uncover idle resources and wasted costs across all your cloud accounts and regions. Consequently, supported resources have nowhere to hide, and there is no way they will slip under the radar. As soon as the resource inventory is fetched, all regions will show exactly what they are holding. The resources come to you in a sense, so there’s no more tab switching or console hoping to make sure you didn’t miss anything.

Having an asset inventory of your AWS resources is crucial to uncover optimization opportunities and answering questions about your infrastructure. AWS has some good services but as the number of resources increases and you shift toward multi-cloud you might want to check out something like Komiser that does it all.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

AWS Security Pillar

LABOUARDY Mohamed — Thu, 23 Feb 2023 21:12:29 +0000

Welcome to this week’s DevOps newsletter! I’ve got a lot of exciting topics to cover, including AWS security pillar, Kubernetes dashboards, and DevOps open-source projects.

Firstly, I have a comprehensive guide to the AWS security pillar, where you can learn how to secure your AWS environment by implementing AWS security best practices and gaining a comprehensive understanding of AWS security services.

Next, we’re exploring the rise of Serverless monoliths and the best practices for running Java apps on Kubernetes. Additionally, I’m sharing an architecture for enforcing RBAC in a cloud storage system and an open-source utility that scans live Kubernetes clusters and reports potential issues with deployed resources and configurations. And, for those interested in Terraform, I have a preparation guide for becoming a Hashicorp Certified Terraform Associate.

Lastly, don’t miss our open-source projects of the week, including a ChatGPT-powered gym workout generator and a CLI that creates screenshots based on terminal command output. And, we’re also diving into why open-source is broken and uncovering the truth about git metrics tools. Stay tuned for all this and more in this week’s DevOps newsletter!

Posts of the week

🔒 “Our guide to the AWS security pillar” — A walkthrough of the AWS Security Pillar with insights into how to manage this vital but often complicated aspect of modern architecture — Read more »

🧐 “Platform Engineering teams done right…” — Three reasons for the platform engineering meme: demand for tools to improve complicated platforms on Kubernetes, marketing by companies with tools to sell, and interest sparked by the Team Topologies book’s definition of how to create/manage Platform Teams — Read more »

💡 “AWS Lambda layers best practices” — This blog post covers AWS Lambda layers basics, the pros, and cons, and recommended best practices — Read more »

📊 “Kubernetes dashboards: everything you need to know” — Kubernetes comes with its own web UI for deploying containerized applications to a cluster using wizards, troubleshooting workloads, and managing cluster resources — known as Dashboard. But there are other open-source options as well — Read more »

🔥 “Become a Hashicorp Certified Terraform Associate — preparation guide” — The post is intended for individuals looking to prepare or take the exam in the future. It covers tips and what you need to know to pass the exam — Read more »

🔑 “An architecture for enforcing RBAC in a cloud storage system” — This article explores a 2016 paper by Garrison et al. that presents an architecture for enforcing access control policies in a cloud storage system — Read more »

🧵 “Best practices for Java apps on Kubernetes” — In this article, you will read about the best practices for running Java apps on Kubernetes. Most of these recommendations will also be valid for other languages — Read more »

🚀 “The rise of the Serverless monoliths” — This post covers the evolution of meta-frameworks (Next.js and Remix) and backend as a service (Supabase, SurrealDB) — Read more »

🐳 “Docker will edit host-based firewall rules for you” — Docker would quietly add a rule to your system’s iptables to allow container port through the firewall — Read more »

Projects of the week

IaSQL is open-source software that treats infrastructure as data by maintaining a 2-way connection between a cloud account and a PostgreSQL database — Learn more »

Signadot is a Kubernetes native platform that provides lightweight environments using a unique multi-tenancy model that shares resources safely. You’re able to test every pull request end-to-end in K8s and ship features 10x faster — Learn more »

Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations — Learn more »

A cool side project that leverages ChatGPT to build gym workouts for you based on the equipment you have at your disposal — Learn more »

Keep is an open-source alerting CLI that contains everything you need to start creating Alerts. It supports all major providers (e.g. Sentry/Datadog or Slack/Pagerduty) — Learn more »

Termshot takes the console output and renders an output image that resembles a user interface window. The idea is similar to what carbon.now.sh, ray.so do — Learn more »

Tweet of the week

If you’ve ever noticed dropped connection after a rolling upgrade, this thread digs into the details👇🏻

Meme of the week

I hope this summary has been helpful. Remember to subscribe to the newsletter to receive the latest DevOps trends in your inbox every week 🔥

Drift management in cloud infrastructure

LABOUARDY Mohamed — Wed, 22 Feb 2023 17:15:31 +0000

Over the past few years, the number of infrastructure services has grown, and more applications are being released to production on a daily basis while infrastructure needs to be able to be spun up, scaled, and taken down frequently. The adoption of CI/CD and DevOps practices emphasizes the importance of having similar runtime environments. Without an Infrastructure as Code (IaC) practice in place, it becomes increasingly difficult to manage the scale of today’s average infrastructure environment.

IaC safeguards the entire process of cloud provisioning and ensures consistency across different environments by codifying and documenting configuration specifications. IaC tools like Terraform helped the dev and ops teams align as they both use the same description of application deployment. In an ideal world, you want everything to be managed by your IaC stack but expectations do not always line up with reality, resources are still being provisioned manually or through the cloud provider’s consoles, causing infrastructure drift and a growing number of untracked assets.

Understanding the resources that are not managed by IaC in the cloud is a challenge and finding whether they remain in the same configuration defined in the code is yet another task. This blog post will explore the various tools available for detecting and managing infrastructure drift.

What’s Infrastructure Drift

Infrastructure drift occurs when the configuration of the infrastructure deviates from its intended or documented state. This deviation can be attributed to several factors such as human error, lack of automation, manual intervention, applications making unwanted changes, changes applied to some environments but not propagated to others, and so on, leading to inconsistencies in the infrastructure. Additionally, CI/CD workflows can result in failed pipelines, causing the infrastructure state to become corrupted, leading to orphaned resources and drift.

One major cause of infrastructure drift is the creation of resources outside of the established IaC tools such as Terraform, CloudFormation, and Pulumi. When this happens, the infrastructure state is not adequately described or persisted, and the changes made to the infrastructure go unnoticed. This opens the door to security vulnerabilities, wasted costs, and compliance issues.

In some cases, there may be production incidents or emergencies that require quick action, and manual adjustments to the infrastructure via web consoles may be necessary to achieve a better state as soon as possible (and keep the customers satisfied). However, this becomes a problem when those changes are not backported to Terraform which often stems from poor education on best IaC practices, loose access permissions, and a lack of proper communication regarding the infrastructure management process.

Why it’s bad

Infrastructure drift can have a significant impact on the reliability, security, and cost-effectiveness of the infrastructure. One major issue caused by infrastructure drift is the wastage of cloud resources, which can lead to increased costs. Drifting can result in the creation of duplicate resources or the failure to delete unused ones, leading to an unoptimized cloud environment.

Furthermore, infrastructure drift can pose a significant threat to the security of the infrastructure. Inconsistent configurations can make the infrastructure vulnerable to security breaches and data leaks. Such inconsistencies can lead to essential resources unintentionally being made publicly accessible, and unsecured resources may go unnoticed. However, if changes to infrastructure were made through IaC, it would be possible to set up compliance policies and security controls, preventing or mitigating issues such as an S3 bucket being accessible to the public and making sure all resources are properly tagged.

Infrastructure fragmentation is another problem that can arise from infrastructure drift. As the infrastructure becomes more complex, it becomes more difficult to track all resources and changes. This can lead to situations where development teams are unaware of production environment changes, which can cause applications to crash and deployment projects to fail unexpectedly. Moreover, when the IaC tool does not cover the entire infrastructure, it can cause discrepancies between the different environments, leading to inconsistent behavior. This inconsistency can be particularly problematic between the development, staging, and production environments.

Without a single, shared source of truth, intentional infrastructure changes to remediate incidents could be reverted or temporary changes left unnoticed, wasting thousands of dollars in monthly costs due to unused resources.

Cloud workloads undergo frequent changes as more workloads and services are deployed to the infrastructure, resulting in more developers and authenticated services interacting with the infrastructure across various cloud environments and providers. Drift is inevitable, just like incidents, and is a part of the infrastructure’s life cycle. Therefore, it’s crucial to be able to easily and quickly detect and possibly revert drift.

Drift Management

Preventing and resolving infrastructure drift is crucial to maintain the stability and security of the infrastructure. Increasing the adoption of IaC is one of the most effective ways to prevent infrastructure drift. Teams should ensure that a greater percentage of the infrastructure is managed by IaC and leverage code versioning, code reviews, static analysis, automated tests, and so on.

When resources are created using IaC tools, drift can be detected and resolved promptly. For instance, running a command like “terraform plan” can reveal any drift in resources described in the Terraform files.

In the screenshot above, we can see that the EC2 instance owner has changed outside of Terraform which is drift.

CloudFormation has a built-in drift detection feature that can be used either via the AWS Console or via the AWS CLI command.

Regular testing and monitoring are also critical to detect and resolve any issues that may arise due to infrastructure drift. Open-source tools like driftctl, terrascan, and cloud custodian can also be leveraged to detect all changes outside of regular IaC workflow and ensure prompt remediation.

In addition to tracking infrastructure changes, it is crucial to track who is provisioning what, where, and how often. This is especially important since it can be challenging to track those changes across multiple cloud providers and accounts, and manually checking provisioned resources can be time-consuming. Tools like Komiser can be used to build a queryable asset inventory and get a clear picture of the cloud infrastructure. Komiser can detect the drift of managed resources and unmanaged resources in multi-cloud environments, which can be brought under control to maintain consistency and prevent security risks.

After loading the cloud assets into the Komiser dashboard, teams can use filters and views to query the inventory and identify any unmanaged resources. This feature enables you to efficiently manage your cloud infrastructure and ensure that all resources are tracked and appropriately accounted for through your IaC workflows.

Cloud Resource Coverage

In addition to the previous points, it is important to regularly schedule drift detection checks to identify any changes that may have occurred. For instance, an hourly check may be appropriate for detecting any changes in IAM roles, while a daily check may suffice for less critical cloud services. Additionally, to minimize the possibility of infrastructure drift due to manual changes, it is recommended to follow the Least Privilege Principle and restrict permissions to cloud practitioners only for necessary tasks. This approach reduces the number of individuals who can make manual changes to the infrastructure.

In summary, preventing infrastructure drift requires a proactive approach, and a combination of practices and tools can be leveraged to achieve this goal. By increasing IaC adoption, regularly testing and monitoring the infrastructure, and leveraging tools like driftctl and Komiser, teams can detect and resolve drift promptly, maintain consistency, and prevent security risks and bill shocks.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

How to practice FinOps with Komiser

LABOUARDY Mohamed — Mon, 20 Feb 2023 16:49:28 +0000

Financial Operations (FinOps) is a critical aspect of cloud computing that helps organizations to manage their cloud resources effectively and efficiently. With the increasing popularity of cloud computing, the importance of FinOps has only increased (94% of enterprises overspend in the cloud), as organizations look to reduce their cloud spend and make the most of their investments in cloud infrastructure.

Despite the increasing concerns around cloud costs, there has yet to be a single tool that comprehensively manages and helps to remediate excessive cloud expenses. As a result, teams continue to depend on a mix of native cloud provider tools, third-party platforms, and Google Sheets.

According to the State of FinOps 2022 report, teams are still struggling with the following challenges:

FinOps is a cultural discipline that involves collaboration among finance, engineering, product, and management. The open-source model can facilitate this collaboration and cover the long tail of cloud providers and services. That’s where Komiser comes in as a way to address cloud cost management concerns in today’s multi-cloud environments. It offers insight into cloud resource consumption and expenses, making it a valuable tool for organizations practicing FinOps.

In this post, we’ll explore how to use Komiser to empower engineers to optimize their cloud spend while following the FinOps principles.

Cost Allocation

Cost allocation is a crucial component of FinOps. Komiser enables cost allocation to individual projects, teams, or departments, which simplifies the tracking of resource usage and its associated expenses. This information can be used to set budgets, track costs, and identify areas where costs can be optimized. For example, if you see that one team is consistently using more EC2 instances than others, you can work with that team to identify opportunities for optimization.

By utilizing a tagging strategy, you can analyze cloud resource utilization and costs based on specific criteria, such as teams, applications, or cost centers. This approach enables the creation of personalized views that promote accountability for cloud expenses.

This is a great way to show your teams what they’re spending and why and see the impact of their actions on the monthly bill.

Multi-cloud Cost Reporting

As you’re moving to a multi-cloud model, you would need a single place where you can manage all your cloud resources. By integrating with several cloud service providers, Komiser can swiftly generate your cloud asset inventory. This allows you to utilize its powerful filter system to uncover idle resources and wasted costs across all your cloud accounts and regions.

Komiser takes cloud cost management to the next level. Firstly, it gives you visibility into the cloud unit economics that are relevant to you, rather than focusing on a specific cloud vendor. Additionally, it enables you to tag resources across multiple providers and regions using a single interface, which can uncover dormant resources and reduce unnecessary costs.

Komiser provides multi-cloud platform support, including AWS, DigitalOcean, OCI, Tencent, Linode, and Civo, with GCP and Azure support to be added soon. It also supports containerization solutions like Kubernetes.

Uncovering Idle Resources

Komiser provides detailed information on resource utilization, including information on the types of resources being used, the number of resources being used, and the costs associated with each resource. This information can be used to identify opportunities for cost optimization, such as underutilized resources or resources that can be scaled down to reduce costs.

By assigning human-readable labels to cloud resources, teams can use tagging to increase visibility and make smarter budget allocations. With Komiser’s bulk tagging feature, tags can be efficiently applied to a group of resources provisioned in various providers and regions without leaving the Komiser dashboard.

With resources being accurately identified by tags, you can gain a comprehensive understanding of your cloud costs, pinpoint resources that are either redundant or aren’t being used, and identify potential opportunities to save money.

Shared Costs Tracking

A key aspect of FinOps is that every team member is responsible for their cloud usage. The calculation of the total cost of ownership requires transparency and accuracy, but unallocated shared costs obstruct these factors. If shared costs are not properly divided, engineers and product managers do not have a complete understanding of the actual cost of their apps. To improve visibility into shared resources such as databases, logging, k8s, enterprise support, etc., developers can categorize shared resources and allocate budget, as well as create custom views that separate these shared resources from the rest of the team’s views.

Cost Trends Analysis

It’s important to keep an eye on cost trends over time. Komiser provides information on cost trends, including total costs, cost per resource, per team, or tags. This information can be used to identify cost spikes or patterns in resource usage that may indicate an opportunity for optimization. For example, if you see a sudden increase in costs for the frontend team, it may be a good time to review your CDN utilization and see if there are any opportunities to reduce costs.

Komiser cost explorers go beyond native tools like AWS Cost Explorer to provide full-funnel cost visibility across your cloud environment.

Budget Monitoring

In order to stay on top of costs and ensure that resources are being used effectively, it’s important to set alerts in Komiser. Slack alerts can be set to notify you when costs reach a certain threshold, or when resources are being used more than expected. This can help you to catch potential cost optimization opportunities early before they become larger problems.

We need your help

Although Komiser can serve as a good starting point for enabling FinOps within your organization, there is still room for improvement in terms of features. We’re collaborating with the open-source community and cloud leaders to work on the following enhancements:

Automatic resource scaling, right-sizing of instances, RI coverage, and other cost optimization recommendations.
Forecasting based on past cloud usage to anticipate future demand and identify areas for investment.
Additional support for cloud providers and SaaS platforms.
Container cost reporting through OpenCost.

Join our Discord community or visit the Komiser repository to find a good first issue and help us create the future for DevOps where the cloud is transparent and collaborative.

Conclusion

Developing a FinOps culture takes time, but it is essential for creating a sustainable business model. With the right strategy and tools, such as Komiser, you can automate cloud cost management, address complex edge cases, and set higher performance goals. By building a cloud asset inventory, tracking usage and costs, and setting custom alerts, organizations can optimize their cloud infrastructure and make the most of their investments.

Whether you’re just starting out with cloud computing or are an experienced user, Komiser is a valuable tool for practicing FinOps and optimizing cloud spend.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.