Forem: Tailwarden

Take back control of your tags with Tailwarden - Part 2

Jake Page — Thu, 21 Dec 2023 08:40:01 +0000

In part 1 we explored the importance of having a clear understanding of where your tagging strategy stands at a given point in time. Is it well enforced? Is it additive to your overall cloud management efforts? We looked at how leveraging Tailwarden features like the tags audit and using inventory filters can open up a first window into the current cloud tagging implementation. In this blog post, we will build on those features and talk about the next phase of the tagging execution strategy. Remediation.

Remediate or start from scratch?

In an ideal world, you would rarely be held down by constraints and could start projects from a clean slate. 100% of your time would be spent on writing new code, and new strategies and processes would never have to fight for priority over legacy ways of doing things. Unfortunately though, as many developers and engineers have noticed when joining a new organization, apart from all the fun work, a non-trivial amount of time goes into maintaining already existing code and infrastructure, it’s just the way things are. If you’re lucky, the original developers had maintainability and code longevity in mind, if not, my sincere condolences.

Applicable also to cloud resource tagging strategies, even though, in many cases, it would be desirable from an implementation point of view to start fresh and adopt a brand new approach, this is not always feasible. And since the main aim of this blog post series is to give practical advice applicable to real-world use cases, I feel it’s more appropriate to focus on remediation over scraping your current tagging strategy and starting over. In many cases, the sheer size of enterprise production environments would involve prohibitive amounts of resources, becoming incredibly expensive and a decision the C-Suite is not likely to go for.

Pass me the salt

Cloud engineers, SREs, and FinOps practitioners, regardless of the role, have a series of objectives. Control costs, security concerns, compliance, etc. None easy tasks, especially if your environment spans multiple cloud accounts or providers. By default, cloud providers don’t offer native features to enable cloud tag unification across cloud providers.

At a recent Tailwarden virtual meetup, Alex Jones brought up an interesting analogy that draws parallels between multi-cloud environments and tables in a restaurant. In this analogy, each cloud platform is likened to a unique table, complete with unique tablecloths and salt shakers. The challenge arises when there is a desire to share resources, represented by the salt shakers, among these disparate tables.

To address this challenge, an abstraction is needed, and Tailwarden serves as that unifying layer. Picture each cloud table being brought a bit closer together, and a larger cloth, symbolizing Tailwarden, is draped over both tables. This larger cloth effectively covers and connects the individual tables, allowing for seamless resource sharing and tag unification. Tailwarden acts as a bridge, offering a common interface that spans multiple cloud platforms and simplifies the complexities associated with managing a multi-cloud environment.

Business units, squads, and internal services don’t always fit neatly into provider-segregated buckets. This being the case it’s essential to have a management layer combined with a comprehensive cloud tagging strategy that acts as the ultimate control panel. So many multi-cloud management tasks are made easier by having this level of abstraction. But how can we tie these different cloud providers together and create a unified organizational view? By using virtual tags.

Don’t have a Tailwarden account yet? Book a demo here and open your free account.

Implement Virtual Tags

Virtual tags are the Tailwarden layer key/value pair that can be easily attached to cloud resources to bridge the gap between cloud providers and to quickly rescue resources that appear to be untracked and unaccounted for.

Easily add missing critical tags.

In situations where the existing tagging implementation isn't necessarily broken but rather incomplete, virtual tags become a valuable tool for filling in the gaps and patching resources that require tracking. While Tailwarden recognizes two types of tags, provider and virtual, when filtering resources by tags, it takes all tags into account.

Virtual tags will not show up in the target cloud provider so as a manager of your cloud resources you are free to make logical groupings without worrying about interfering with conflicting with tags created by other stakeholders.

Add cross-provider tags

Virtual tags empower you to construct an interconnected tapestry that aligns with your business units, squads, and overall service architecture. When dealing with a service that spans multiple accounts or cloud providers, maintaining visibility over all pertinent resources has never been more accessible and straightforward.

Routinely Reference The Tags Audit

Treat the tags audit widget as the source of truth of your current tags implementation. Be it Virtual or Provider-level tags, easily spot inconsistencies, duplicates, or non-compliant tags. The feature makes it easy to click and zero into the resource to quickly find it inside the cloud provider itself.

Recommended workflow:

Tag your resources in the cloud provider or on Tailwarden using Virtual tags
Check the Tags Audit widget to see the current state of your cloud tags
Access resources with the same tags applied by clicking on the tags
Need to update a resource? Easily access the cloud provider through Tailwarden

Access The Cloud Provider Console In Just One Click

Remediation is preferable over starting from scratch only if it is quicker and easier to execute. It’s crucial that very few obstacles stand in your way from the time to spot an inconsistency to the time it takes to address it. Tailwarden facilitates this process through the resource details view, offering a direct link to the resource in the cloud provider. This enables you to promptly update tags, address naming inconsistencies, or take any other necessary actions directly on the resource.

Don't delay obtaining key insights into your environment waiting for the perfect tagging strategy. Take swift action by updating your cloud resource tags directly on Tailwarden using both provider and virtual tags. Ensure all your business units, squads, services, and environments are thoroughly represented through tags and easily filterable.

Regardless of how long it takes to remediate your current situation, once you've established the desired tags, it's crucial to maintain consistency. But how can you effortlessly monitor and ensure your cloud tags remain in compliance? This topic will be explored in the final installment of this tagging blog post series.

Take back control of your tags with Tailwarden - Part 1

Jake Page — Mon, 18 Dec 2023 15:02:35 +0000

Having a strong and consistent cloud tagging strategy is one of the basic building blocks needed for any organization that is seriously trying to effectively manage and thrive in the cloud. But even a great strategy if not accompanied by excellent execution, won’t be very effective. That’s why I want to take a practical approach to exploring some Tailwarden features that can aid in effective tagging management. In the following three articles, I will do my best to show how a well-executed strategy can make all the difference in your cloud management efforts.

A tag is one of the easiest cloud concepts to explain, a key/value pair attached to a cloud resource holding any information you want. A well-defined tagging strategy that's robust yet flexible, well-implemented, auditable, and future-proof shouldn’t be much more difficult. However, it starts getting complicated if you don’t have clear answers to questions like, who is in charge of the tagging strategy? When should tags be attached to a resource? Are there any organization-wide mandatory tags? Can team-level tags be added? Are tags case-sensitive? Are there any enforcement policies in place?

Having a tagging strategy isn’t optional. If you have untagged resources in your environment, it doesn’t mean you lack a tagging strategy it just means your strategy leaves a lot to be desired. Think of it this way, just because your car doesn’t have any license plates on the bumpers, doesn’t mean it’s not registered to you. Once this is internalized we can start challenging some assumptions, and you can start focusing on consistently executing, remediating, and evolving the tagging strategy.

💡 Don’t have a Tailwarden account yet? Book a demo here and open your free account.

The Downsides of a Bad Strategy

Unless your strategy is rock solid you can never be sure you are not leaving money on the table

If tags are sparsely implemented or not carefully maintained across teams and departments. When you gather infrastructure cost metrics from aggregation tools like AWS Cost Explorer or Tailwarden you can never be sure to have 100% coverage. By having inconsistent tags, crucial resources might be left out of budgets but you can be sure that they will appear in the cloud provider's monthly bills.

Unless you have an owner you don’t have a strategy.

A tagging strategy is not an organic process that self-preserves and improves automatically. It needs an individual or team to take responsibility for it and commit to maintaining and ensuring enforcement across the board. If not, it can lead to being a hindrance to future management efforts and a huge missed opportunity. What does this mean in practice? Apart from every resource having an Owner or Team tag, there should be a person or team who is responsible for company-wide tagging compliance, making sure all Owners and all Teams are in alignment with the larger strategy.

It’s impossible to have a cost-conscious culture at your organization without hard and reliable data.

A consistent tagging strategy is one of the main ingredients necessary to be able to consistently generate reliable, granular, and reproducible resource usage data and cloud transparency metrics. If one of your objectives is to nurture a culture of cost consciousness and resource safeguarding, it has to start with a reliable source of truth.

I assume you already have a tagging strategy in place for better or worse and you are not starting a green-field project. In the upcoming blog posts, we will look at practical advice that can be applied to your organization immediately. We will break down our exploration into three execution phases, starting by Evaluating your situation, then Remediating the Situation, and finally, Ensuring consistency over time as well as showing how some Tailwarden features can be utilized to make managing your cloud tags much easier.

Understand your current implementation

Who owns and executes the tagging strategy?

This is a crucial question you have a clear answer to. Tags are useful at every level and since cloud providers can hold up to 50 or 60 tags per resource in some cases, individual contributors; should be encouraged to use custom tags that help them in their work. But there has to be an overarching cohesive system of communally agreed upon tags (a tagging strategy) that tools like Tailwardencan leverage to glean high-level, organizational insights. The core organization tagging requirements have to be implemented top-down and monitored across namespace, account, and provider in a centralized manner.

Which tags are company-wide and mandatory?

There is no “silver bullet” industry standard tagging strategy, each company is different and need not copy what other companies do. Having said that, inside the organization itself, there must be no doubt at all about which tags must be applied company-wide and preferably at creation time.For example, some commonly used tag keys are:

Team: To what squat does the resource belong?
CostCenter: Whose budget does the resource fall into?
Environment: Production, Staging, Sandbox, TestingAccount
Service: Does this resource belong to a larger service cluster?

How consistently have the tags been applied?

Understandably as environments and teams scale, maintaining tagging compliance can become a difficult task. If you are using some sort of IaC tool you might be able to ensure compliance at creation time but how do you know tags weren’t edited or removed afterward? This is where some tooling can help.

Tags Audit feature

Once you connect your cloud accounts to Tailwarden your cloud resources will be automatically fetched, and along with them any tags attached to the resources will also automatically appear.

To inspect your currently implemented tags navigate to the tags audit widget, found in the main dashboard which displays the tag key and value, the number of instances of the tag as well as the stage coverage rate, which equates to the percentage of resources that have that particular tag attached.

Tailwarden distinguishes between two types of tags. Provider tags are created at the cloud provider level. Virtual tags on the other hand are tags created inside Tailwarden. The cloud provider is unaware of the existence of virtual tags but they can be very useful for you to extend business units, services, and apartments across multiple cloud accounts or even providers.

Tags audit gives you quick access to the resources and with just a couple of clicks, you can find them inside your cloud account.

The Tags Audit widget acts as the first line of defense to understand the current state of tagging in your environment. Quickly catch spelling mistakes, and repeated or underutilized tags.

What insights are readily available?

Along with the tags audit feature, gain further insight into your tags by applying specific filters to the resource inventory to get answers to questions like, do I have any untagged resources? Are all of the most expensive resources correctly tagged? Are all mission-critical resources tagged with all required tags?

Untagged Resources

The lowest-hanging fruit in trying to understand the current tagging implementation is to filter for all empty tags. In the example below all Untagged AWS resources which were created in the last 7 days will appear. Any resource that appears in this search will violate the tagging strategy and should be addressed.

Finding expensive critical resources

The specific tag filter combined with a Cost greater than 100 dollars filter shows 0 resources. Good news right? Are we sure that all costly resources have the current Environment:Prod tag? Or are they even tagged at all?

Remove the tag filter and compare the results, are there any costly resources that are untagged? If so, fix it.

Easily find missing tags

Filter specific categories like service or region and add the Specific tag / Doesn't exist to generate a list of all resources that are missing a key tag.

In this case, I have 232 IAM roles that don’t have an Owner tag attached to them. I have some tagging to do.

Save your most crucial filters as Custom Views

As you leverage tags to filter the inventory list, save the results as Custom Views. When you filter the inventory list by including tags you are effectively shining a light on deep dark and unvisited areas of your infrastructure you might have never seen before. Make sure you have quick access to your environment by saving a granular filter as a custom view to always have quick access to key infrastructure insights.

Alerts and filters can also be applied to custom views themselves but we will cover that in more detail in the blog posts to come.

As the cloud, in IT and business management in general, I would not be the first one to assert that an incredibly sophisticated tagging strategy isn’t worth very much unless it is executed diligently. Without regular checks and the right tools to ensure consistent implementation and prevent divergence, even the most intricate strategy can lose its effectiveness. When your implementation isn’t perfect, it’s time to remediate. An integral phase of effective tagging execution and also the topic of the upcoming blog post.

Overcoming Anchoring Bias in Cloud Cost Management

Jake Page — Thu, 24 Aug 2023 14:10:25 +0000

Introduction

I recently picked up a book given to me as part of my Tailwarden onboarding welcome kit. It came with sizeable boots to fill since our CEO scribbled a note on the inside cover saying it was one of the most fascinating books he had ever read, I had to see for myself.

He wasn’t wrong, "You are about to make a terrible mistake" explores the fascinating instances of how cognitive biases distort our decision-making capabilities. Countless examples illustrate how our implicit biases affect us all, gaining awareness of their presence is the first step towards anything resembling an antidote. "Confirmation bias", "Storytelling", "hindsight" and "present" bias to name just a few invisible obstacles we are up against. There are so many ways our intuitions can fail us, it’s a miracle we can ever get anything done in the first place. Granted there are a multitude of biases that we could explore in this article, but in the context of cloud cost optimization and FinOps there is one bias I found to be particularly salient. A bias that if taken into account can unlock eye-watering cloud resource savings. We are talking about the Anchoring bias.

What is Anchoring bias anyway?

It’s a commonly occurring bias that emerges when individuals rely too heavily on the first piece of information they receive, using it as a reference point or "anchor" for making subsequent judgments or estimations. This initial information, even if irrelevant or arbitrary, can have a powerful influence on their final decisions.

For example, let's say you are looking to buy a new laptop, and the first one you see has a very high price tag. Even if it's not the best laptop for your needs, that initial high price might become your anchor. As you continue to explore other options, you might unconsciously compare their prices to the initial high price, perceiving them as cheaper or more expensive relative to that anchor. As a result, you might end up making a purchase decision based on the anchoring bias, even though there might be more suitable and reasonably priced options available.

In the context of cloud cost optimization and FinOps, if a team sets a high initial budget for a project, they might be anchored to that figure and end up overspending on cloud resources throughout the project's lifecycle.

Recognizing anchoring bias and being mindful of its influence can help individuals and teams make more objective and data-driven decisions, ensuring that they don't get unduly swayed by initial, arbitrary, or irrelevant information.

The Pitfalls of Anchoring Bias in Cloud Optimization

Settling for modest cost reductions due to fixation on initial data

Many times in complex cloud environments you are going to encounter large and many times hard to understand bills. In most cases, a few services have an outsized weight on the overall bill, such as compute instances, databases, or complex managed services. The main cost savings opportunities will involve these services. However, here's the catch: as you delve into your cost-cutting endeavors and pinpoint the exact expenses for each service and resource, there's a risk. You might inadvertently anchor your focus on these initial cost figures. Consequently, the danger lies in settling for reductions that, while certainly noteworthy compared to the starting costs, might pale in comparison to the much larger savings potential that could realistically be achieved. In other words, you might be leaving “money on the table”. This fixation on the anchor values blinds you to the more significant optimization possibilities that await deeper exploration.

Overlooking significant savings opportunities in the long run

The risk of narrowly trying to reduce the cost of the resources that are provisioned may lead to larger overhauling and re-structuring efforts that might lead to monumental savings being overlooked. The danger is that you focus too much on the tree and miss the forest. For example, you might only ask questions like, how can I get a 10% reduction of that compute instance? and not questions like, is this the right instance type for the job, why are we not using spot or reserved instances instead? Are my compute instances scaling horizontally in an efficient manner? How long do we actually need the Cloudwatch logs to be persisted? Is serverless an option for the use case?

The impact of anchoring bias on strategic cloud resource allocation

You will find that anchoring bias greatly influences the questions you ask. And during any strategic planning effort, it's essential to frame the conversation correctly and base it on premises that can set you up to win. For many managers and divisions inside an organization, the budget allocated to the team is a proxy for how much the team is valued in the company. This leads to incentives that align with budget conservation. For those who tie their teams identity to the budget that is allocated to them results in very little variance from quarter to quarter. To be the person in the budget meeting to propose slashing the budget in half can win you foul looks from colleagues. This has to be completely thrown out of the window when in the context of cloud resource cost allocation.

Examples of Anchoring Bias in Cloud Management

Case Study 1: E-Commerce Scale-Up

Scenario: An e-commerce company experiences a surge in website traffic and subsequently sees a significant spike in its cloud expenses, particularly related to database read/write operations.

Anchoring Bias: The DevOps team fixates on the initial data that indicates database operations as the primary cost driver. They focus their efforts solely on optimizing database-related expenses.

Impact: The DevOps team successfully reduces database costs by 10% through query optimization. However, they fail to notice that a considerable portion of the expenses now comes from auto-scaling server instances required to handle the increased traffic. They also fail to ask themselves if the current database architecture is still appropriate if similar surges are to be expected.

Case Study 2: Media Streaming Platform

Scenario: A media streaming company notices a gradual increase in its cloud expenditures due to higher demand for streaming content. They identify that a specific type of storage service, used for storing user data and media files, is the costliest component.

Anchoring Bias: The operations team becomes anchored to the initial data that points towards the expensive storage service as the main issue. They dedicate their efforts to optimizing the usage and pricing of this storage service.

Impact: The operations team indeed manages to optimize the storage service, achieving a 20% reduction in costs. However, during this process, they overlook the fact that a significant portion of costs are attributed to transcoding servers responsible for converting media files into various formats. The team's fixation on the initial anchor prevents them from considering a more extensive overhaul of the transcoding architecture, which might involve adopting more efficient codecs or utilizing specialized hardware. As a result, the potential for substantial savings from a more significant transformation remains untapped, highlighting how anchoring bias can lead to suboptimal decisions.

Combating Anchoring Bias with Tailwarden

Awareness is half the battle

Awareness of the anchoring bias within us marks a crucial starting point. Effective cloud cost reduction strategies thrive on data-driven decision-making, incorporating various metrics. By expanding your analysis beyond singular resource costs, you create space for robust choices. Tailwarden's Cost Explorer feature aids in revealing historical cost trends, highlighting resource-heavy areas. Yet, it's not only about understanding high costs, it's about contextualizing their role in your architecture. Regular evaluations of alignment with budgetary needs are key. Tailwarden cloud agnostic resource management features can act as your guide, providing insights for well-informed decisions.

Collaborative decision-making to counter individual biases

One of the main takeaways from the book is that from an individuals perspective it's extremely hard if not impossible to be 100% aware and consious of ones biases at all times. It's even excruciatingly hard to be aware of ones biases at all for that matter. The only way to counteract the biases of the individual decision maker is to share the load, focus on the process itself of making decisions and engaging in dialogue, a lot of dialogue.

Olivier Sibony, the author of the aforementioned book, raises concerns about other biases surfacing within group dynamics. Topics we can explore in another article, include Groupthink, Information cascade, and Group polarization. What's evident, however, is that a group united by the goal of becoming a data-driven entity, unafraid to question past strategies and rally behind bold cost-cutting aspirations isn't just the best way to counteract biases, it's pretty much the only way.

Regularly update assumptions and strategic plans

As you use Tailwarden to gain deeper insight into the inner workings and behavior of your cloud environments, use the data points and granular cost data to inform regular conversations and working sessions with the explicit objective of challenging assumptions about how your budget is structured and how it is currently allocated. Build internal processes that help make the best decisions, use clear and concise data to inform the group conversation, encourage dissenting voices and set a deadline for a decision to be made.

Conclusion

Simply put, if anchoring bias goes unchecked it can hold you back and deny you major cost savings in the long run. The stakes are usually quite high, with ever-growing cloud environments, cloud bills quickly stack up and become the driving expense of many companies. As cloud engineers and developers, you have an outsized role in the impact on your company's bottom line. By maintaining a clear focus on the importance of unbiased cloud decision-making, and always remembering the potential that any single decision maker has to be a biased single point of failure, you are setting yourself and your team up for long-term cost savings and efficiency.

Deploy Komiser to AWS with Terraform

LABOUARDY Mohamed — Mon, 24 Apr 2023 18:57:57 +0000

Komiser is an open-source cloud-agnostic inventory solution that gives you a holistic view of your cloud resources and helps you uncover insights about your infrastructure such as cost wasted, security threats, and compliance issues.

This tutorial will cover all the necessary steps for deploying Komiser on AWS using Terraform. By the end of this tutorial, we’ll deploy the below architecture with the following components:

Provision an EC2 instance to host the Komiser container, and ensure that inbound and outbound traffic to the instance is restricted using a security group.
Deploy an ELB in front of the EC2 instance to manage traffic distribution, and configure a security group for the ELB.
Create an IAM instance profile and attach it to the EC2 instance, including the Komiser recommended IAM policy to manage access to AWS resources.
Set up an alias record in Route 53 to direct traffic to the ELB DNS with a user-friendly URL.

All Terraform templates used in this tutorial can be found in the GitHub repository.

To get started, define your backend and declare AWS as your provider in the terraform.tf file. In this example, S3 is used as the backend for storing Terraform state files. Once done, run terraform init to download the AWS module.

Next, declare an EC2 instance in ec2.tf file with aws_instance resource. The resource uses Amazon Linux 2 as an AMI which is obtained using the data block and the aws_ami data source. The instance type is t2.medium (recommended size to host Komiser) and uses a public IP address and a security group that allows traffic on port 22 for SSH access and 3000 for serving the Komiser dashboard. It also attaches an IAM instance profile with the permissions required by Komiser to build your asset inventory.

The provisioned blocks define a series of file transfers and commands to execute on the EC2 instance after it’s launched. The first three provisioner blocks upload files from the local machine to the EC2 instance. The last provisioner block executes remote commands on the EC2 instance, which installs some needed dependencies by running a bash script that is transferred to one of the previous provisioner blocks and deploys Komiser as a Docker container.

It’s important to note that when using Komiser in a production environment, certain additional improvements should be taken to ensure security and scalability.

Firstly, it’s recommended to deploy the Komiser instance within a private subnet and restrict SSH access only from a trusted CIDR block or a bastion host. This helps to prevent unauthorized access to the instance and reduces the risk of security threats.

Secondly, automation tools such as Ansible or Packer can be leveraged to further optimize the deployment process. Using Packer, for example, allows for creating a custom AMI that includes all the necessary software and configurations for running Komiser.

The install.sh script installs Docker Community Edition (CE) and Docker Compose. It also adds the ec2-user to the docker group, allowing us to run Docker commands without needing to use sudo.

Once those tools are installed, the deployment of the Komiser container can be initiated by executing the command docker-compose up with reference to the docker-compose.yml file. This will result in the deployment of the latest version of the Komiser image, which is at the time of writing this post v3.0.11.

The container is configured to use a config.toml file that connects to the running AWS account and stores data in an SQLite database. Additionally, the container serves the Komiser dashboard through port 3000.

In the iam.tf file, the next step is to define AWS IAM resources that enable the Komiser EC2 instance to assume an IAM role with the appropriate permissions attached to it.

Next, in elb.tf define a load balancer resource that forwards traffic into the EC2 instance running the Komiser container on port 3000. The ELB is configured with two listeners: one for HTTPS traffic and the other for HTTP traffic. The HTTPS listener is configured with an SSL certificate (requested through ACM) that is specified as a variable.

The health_check block specifies a health check configuration for the ELB. It specifies the target to check for health as “TCP:3000”. The health check is configured to check the instances every 5 seconds, with a timeout of 3 seconds, and a threshold of 2 checks for both healthy and unhealthy responses.

Finally, in route53.tf creates a new AWS Route 53 record for a domain name that points to the ELB resource using an alias record. The record type specified is “A” for an IPv4 address.

This creates a Route 53 record that maps the domain name to the ELB, making it possible for users to access Komiser running on the ELB through a user-friendly URL.

After defining variables and outputs, running terraform plan will generate an execution plan detailing the changes that will be made to the infrastructure. Running terraform apply will apply these changes, resulting in the deployment of the 9 new resources necessary for running Komiser on AWS.

After the resources have been successfully provisioned, you can easily access the Komiser dashboard by navigating to https://demo.domain.com. Once accessed, you will be presented with a comprehensive breakdown of your AWS resources, including their associated costs.

Congrats! You’ve successfully deployed Komiser with Terraform.

You can now leverage Komiser’s holistic view to take control of your cloud usage and optimize your resources for maximum efficiency and cost savings.

Build a Serverless Gym App with ChatGPT, Twilio and WhatsApp

LABOUARDY Mohamed — Mon, 03 Apr 2023 17:05:42 +0000

We have recently started hosting virtual workshops in our Discord community called “Wardens Assembly”. These monthly events cover a variety of tech topics. The first event was about building a Serverless gym app that sends a workout plan to your WhatsApp number using ChatGPT. In case you missed the event, you can watch it completely here:

In case you want to cut straight to the chase, this tutorial is for you as it covers only the key parts of building a Serverless app, including:

Building a Serverless app using Golang & AWS Lambda
Scheduling Lambda with cron expression triggers
Integrating ChatGPT 4 with AWS Lambda
Sending WhatsApp messages via the Twilio SDK
Streamlining deployment with GitHub Actions & AWS SAM

Before jumping into the code, the diagram below summarizes the architecture we’re going to build by end of this tutorial:

The goal is to create a Lambda function in Go, which will communicate with ChatGPT. This function will send a prompt to ChatGPT API and then use Twilio to send a workout plan to our WhatsApp number at a specific schedule determined by an EventBridge Rule. Moreover, we will leverage AWS SAM and GitHub Actions to automate the infrastructure build and deployment, as well as the CI/CD pipeline.

You can find all the source code used in this tutorial on GitHub.

Building Lambda Function

To get started, from your terminal create a main.go file, initialize a go project, and install the AWS Lambda package with the following commands:

go mod init workout-generator go get github.com/aws/aws-lambda-go

Next, declare the handler function in main.go. The main function calls the lambda handler by calling the lambda.Start method_._

Next, to integrate with OpenAI, you will need to download the OpenAI Go wrapper library:

go get github.com/sashabaranov/go-openai

Then, update the handler, and create an OpenAI client by passing your OpenAI token. Next, start a chat using the CreateChatCompletion method and pass a prompt, and setGPT3Dot5Turbo as the target model (which is the underlying name for ChatGPT). The library also supports other models, such as ChatGPT, GPT-4, DALL·E 2, and Whisper.

The OPENAI_TOKEN value can be generated from the OpenAI platform. Make sure to save the key in a safe place, as it will only be shown to you once.

To send the workout plan to our WhatsApp number, we need to integrate Twilio. To do so, you will need to sign up for a Twilio account, sign in to your existing account, and activate the Twilio Sandbox for WhatsApp. Follow the steps below:

Sign up for a free Twilio account.
Activate the Twilio Sandbox for WhatsApp.
Select a number from the available sandbox numbers to activate your sandbox.
Send join to your Sandbox number in WhatsApp to join your Sandbox.

After sending the message, Twilio should reply with a confirmation message, as shown in the screenshot below:

Now to integrate with our app, install the Twilio Go package with the following command:

go get https://github.com/twilio/twilio-go

Next, add the following code snippet to the Lambda handler. It creates a Twilio client by passing the credentials as environment variables. The Account SID and Auth Token can be found here and should be set as the values for the environment variables TWILIO_USERNAME and TWILIO_PASSWORD respectively. Finally, it uses the CreateMessage method to send the workout plan generated by ChatGPT.

That’s it. Our function handler is ready to be deployed to AWS Lambda!

Deploying Serverless Stack with SAM

For the deployment part, we’re going to use AWS Serverless Application Model (SAM). Once you’ve installed the SAM CLI, create a template.yml that declares a Lambda function called “WorkoutGenerator”, including its source code location, handler function, runtime environment, memory size, and timeout duration.

The template also defines a set of environment variables for the function. These variables are resolved using the AWS Systems Manager Parameter Store, which is a service that stores secure strings and parameters. The variables specified in this template include the OpenAI token, Twilio account credentials, and phone numbers for sending and receiving WhatsApp messages.

Next, use the AWS SAM CLI to build the application and prepare for deployment by running the sam build command. Finally, run the sam deploy — guided command to deploy the AWS resources by provisioning an AWS CloudFormation stack:

The Lambda function is now deployed and running in the AWS Cloud! You can test it out, by triggering the Lambda function manually from the AWS Console. A WhatsApp message should be received with a workout plan generated by ChatGPT as follows:

However, our goal is to have our workout plan generated automatically, ideally before we begin our gym workout. To achieve this, we need to trigger our Lambda function at a specific schedule. We can use an EventBridge rule to define a cron expression that invokes our function every weekday at 7 PM. Update the SAM template below and run it again:

If you head back to the Lambda dashboard, you should see an EventBridge Rule trigger, as shown in the screenshot below:

With our application being completed, let’s build a CI/CD pipeline to automate the deployment process through GitHub Actions.

Defining a CI/CD Pipeline with GitHub Actions

Once your source code is pushed to a remote GitHub repository, create a release.yml file under the .github/workflows folder with the following steps:

Check out the repository under $GITHUB_WORKSPACE, so the workflow can access it.
Set up Python environment and install SAM CLI.
Configure AWS credentials from secrets. Make sure to add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as a secret under the GitHub repository.
Run the sam build and sam deploy commands in a non-interactive mode.

Once the pipeline is defined, push the changes to the remote repository.

We can test out the pipeline by improving the ChatGPT prompt to generate a workout plan with only weightlifting exercises. Push the changes, the pipeline will be triggered and new changes will be deployed to AWS as shown below:

To improve the workout plan, provide your age, gender, weight, height, and one rep max in the ChatGPT prompt.

Congratulations! You have successfully built a personal gym workout generator. We would love to see your implementation. Create a similar Serverless app that integrates with ChatGPT and share it on Twitter to enter the competition. Tag Tailwarden on Twitter and add the hashtag #WardensAssemblyChallenge. The three winners will be chosen on Friday, April 28th.

Unexpected charges on your AWS bill

LABOUARDY Mohamed — Mon, 27 Mar 2023 13:36:16 +0000

Are you a new AWS user who has experienced bill shock while being in a free-tier plan or a professional cloud practitioner struggling to understand your team’s cloud expenses? If so, you’re not alone. Many AWS users are surprised by unexpected charges on their monthly bills, which can significantly increase the overall cost of using AWS services. As companies’ cloud environments become more complex, lack of visibility can lead to unpredictable cloud bills and budget overruns. In this blog post, we will cover the most common unexpected charges on your AWS bill and provide tips on how to avoid them, so you can optimize your cloud costs and avoid unpleasant surprises.

It’s worth mentioning that the list is not exhaustive and covers only the most common ways in which money is wasted on AWS.

Paying for unused resources

One of the main common unexpected charges is related to idle resources. AWS charges for resources that are not actively used, such as idle EC2 instances, RDS databases, and Elastic Load Balancers. These charges can accumulate over time and result in significant bills. To avoid this, you can use AWS Auto Scaling to automatically adjust the number of resources based on demand. You can also use AWS Reserved Instances or Savings Plans to reduce costs by committing to a one-year or three-year term or use Spot Instances for non-critical jobs or workloads.

Another common source of idle resources is unattached EBS volumes and unused Elastic IPs. To avoid these charges appearing on your bill, create a policy that automatically deletes any unused EBS volumes or EIPs.

Infrastructure Drift

One major cause of infrastructure drift is the creation of resources outside of the established IaC tools such as Terraform, CloudFormation, and Pulumi, or without proper approval. When this happens, the infrastructure state is not adequately described or persisted, and the changes made to the infrastructure go unnoticed (aka shadow IT activity).

Until you have total visibility across your environment and have implemented measures to prevent the use of cloud consoles, infrastructure drift is likely to contribute to your AWS bill.

Data requests, transfers, and retrievals

Another unexpected item that can pop up on your AWS bill is related to data transfer. AWS charges for data transfer within the platform, as well as data transfer to and from the internet. Many users are unaware of this charge and end up with significant increases. To avoid this, you can use AWS services in the same region or availability zone, which is usually free of charge. You can also use AWS Direct Connect for data transfer between your data center and AWS, which can significantly reduce data transfer charges.

CloudWatch Logs

CloudWatch is the primary source of truth for monitoring the overall health and storing logs of active AWS cloud services. However, it is also notorious for surprise bill spikes due to the complexity of its pricing model.

The pricing is determined by various factors, such as the number of custom metrics, alarms, and dashboards, logs ingested, stored, and analyzed, and the use of contributor insights rules and synthetics canary runs.

The most common way of rapidly driving up costs is by leaving the default retention period. This is especially true for AWS Lambda, which creates an automatic log group with an indefinite retention setting. It’s also important to use alarms and dashboards for key metrics only that way avoiding unnecessary alerts and visualizations.

AWS Free Tier expired or usage exceeds

AWS is pretty generous with free-tier plans but without proper monitoring, you can exceed the free usage limits in a breeze. The good news is you can monitor usage through the AWS Management Console and track free tier usage. Any usage beyond the free tier limit or after a free trial has ended is charged at standard rates. To avoid charges, set up alerts to notify you before the free tier expires or usage exceeds the limit.

AWS Free Tier usage alerts automatically notify you over email when you exceed 85 percent of your Free Tier limit for each service.

Underutilized Reserved Instances and Savings Plans

When it comes to AWS EC2 costs, there are several recommendations that you can use to save money. One popular approach is to purchase Reserved Instances and Savings Plans. By doing so, companies can potentially save a lot of money on their monthly bills. However, it’s important to note that simply purchasing these plans isn’t enough. In order to fully reap the benefits of Reserved Instances and Savings Plans, you need to make sure that they are being used, monitored, and optimized effectively. Failure to do so can result in unexpected charges and higher costs.

Dynamic Environments

Elastic Beanstalk is designed to ensure that all necessary resources are running. As a result, it will automatically relaunch any services that you stop. To prevent this, you must terminate your Elastic Beanstalk environment before terminating any resources that Elastic Beanstalk has created.

Similarly, auto-scaling groups are designed to maintain a minimum number of EC2 instances running. Ensure that you terminate your ASG or update the scaling policies to avoid unexpected charges.

Preventing AWS Bill Shock

AWS has many services to help monitor billing. Setting an account-wide budget alert is a relatively easy first line of defense. Secondly, regularly review and tag your resources to identify any unused or idle resources that you can terminate or downsize to reduce costs. Thirdly, use AWS Cost Explorer to analyze and visualize your costs, identify cost trends, and optimize your spending. Finally, take advantage of AWS tools such as CloudWatch and AWS Config to monitor and optimize your resources continuously.

If you’re looking for an all-in-one platform, you can also leverage open-source tools like Komiser to build your cloud asset inventory, tag resources, set up budget alerts, and bring accountability to cloud spend.

Conclusion

Unexpected charges on AWS bills can put businesses out of the market. However, with the right practices and tools in place, companies can detect and troubleshoot overspending issues before they ever occur.

Whether you’re just starting out with AWS or you’re a seasoned DevOps engineer, Komiser can help you catch potential cost optimization opportunities early, before they become a larger problem.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

Manage your OCI resources with Komiser

Jake Page — Fri, 10 Mar 2023 10:42:42 +0000

In today's digital world, cloud computing has become an essential tool for businesses of all sizes. Cloud computing allows organizations to access a wide range of computing resources, including storage, networking, security, and databases, without the need for on-premises infrastructure.

One cloud provider that has gained a lot of attention in recent years is Oracle Cloud Infrastructure (OCI). OCI is a powerful cloud computing platform that offers a wide range of services to help businesses modernize their IT infrastructure and improve their operations.

Why Oracle Cloud Infrastructure (OCI)?

Oracle Cloud Infrastructure is a cloud computing platform that offers a range of services, including compute, storage, networking, security, and database services. OCI is built on a high-performance architecture that is designed to support modern workloads, including big data analytics, artificial intelligence, and machine learning.

OCI is a reliable, scalable, and secure cloud solution that offers a range of benefits to users. These benefits include:

High Performance

OCI offers high-performance computing resources, including fast network connectivity, low-latency storage, and high-speed compute instances. This makes it ideal for running resource-intensive applications that require fast processing and high bandwidth.

Security

OCI is built with advanced security features to protect against cyber threats and attacks. The platform provides multiple layers of security, including network isolation, encryption, and identity and access management. This ensures that your data and applications are always protected, even in the event of a security breach.

Scalability

OCI offers flexible and scalable infrastructure that can grow and evolve with your business needs. You can quickly scale up or down your resources to meet changing demand, without worrying about hardware limitations. This makes OCI an ideal cloud solution for businesses that need to rapidly scale their operations.

Cost-Effective

OCI offers a pay-as-you-go pricing model, which means you only pay for what you use. This can help you save money on infrastructure costs, without compromising on performance or reliability. Additionally, OCI offers a range of pricing options, including monthly and annual subscriptions, so you can choose the pricing model that best suits your business needs.

Hybrid Cloud

OCI allows you to seamlessly integrate with on-premises infrastructure or other cloud providers, enabling you to build a hybrid cloud environment that meets your specific business needs. This makes it easy to manage your entire IT infrastructure from a single location, regardless of where your resources are located.

As with any cloud provider, it can be easy to lose track of spending, and as your team grows, so does your cloud bill. Fortunately, Komiser has now integrated with OCI, giving you a clear view into your cloud infrastructure. With Komiser, you can easily track, organize, and set alerts for your cloud infrastructure activity, making it much easier to manage and be a responsible cloud custodian. To learn how to set up the integration, follow the steps below.

Configuration

Use an OCI user and an API key for authentication. In this case, you’ll need to put your tenancy OCID, user OCID, region name, the path to an API key, and the fingerprint of the API key.

The easiest way is to let OCI walk you through the setup process by executing the following command using the OCI cli:

Download OCI CLI, find instructions here

Now, a config file and key pair have been created in your local .oci folder:

Since Komiser looks for credentials in a file called credentials. Create a credentials file in the .oci folder and copy the contents of the config file to it.

Contents of credentials file

Add it to the config.toml file like this:

Make sure to update the path to ensure that Komiser has access the credentials file.

Resource inventory view

This is the view of the OCI resources in the Komiser dashboard which has been saved in the OCI Resources custom view.

Supported resources

At the time of writing (Komiser v3.0.5) the supported resources for OCI are:

Compute instances
Developer Service applications and functions
IAM policies
Oracle Autonomous Database
Block storage and buckets

If you're an open source contributor looking to contribute to the Komiser project, there are two ways to do so. First, you can open individual issues requesting support for OCI resources that you'd like to see added to Komiser. These issues can be added to the project by visiting this page.

Alternatively, you can add new resources yourself by using the already supported resources as a template. To get started, take a look at the IAM policy integration example.

For more detailed guidance on contributing to Komiser, check out this guide.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

How to find all resources in an AWS account

LABOUARDY Mohamed — Fri, 03 Mar 2023 17:02:01 +0000

When managing your cloud infrastructure on AWS, it’s important to have a comprehensive understanding of all the resources running in your AWS accounts. It’s crucial to be able to have reliable data and clear insight into areas such as:

Identifying resources that are idle, unmonitored, or exposed to security threats.
Understanding the cost breakdown and coverage of tags.
Keeping up-to-date and accurate audit information.
Evaluating whether your resources conform to specific governance controls.
Detecting any infrastructure drift and changes in configurations.

Without answers to those questions, you open the doors to cost wastage, security threats, and compliance issues.

This blog post provides guidance on the different tools available that can help you in locating and identifying all resources within your AWS account.

AWS Native Services

AWS provides several tools to help you identify and track resources in your account. Each tool has its own pros and cons, as we will see. The key distinction for cloud resource tracking is the scope of the tool and which resources are in its zone. Let’s dive in.

AWS Management Console

For anyone working with AWS, the AWS Management Console is a good place to begin. It provides access to a wide range of services and features. However, the console’s UI can be challenging to navigate, and it can be overwhelming. Additionally, you should have prior knowledge of the resources you are searching for and their location in the region. Otherwise, you may end up spending several hours browsing through multiple tabs and levels of hierarchy in the AWS Console to find answers to questions such as “What is the number of EC2 instances operating in our Frankfurt region?”

AWS Resource Groups

AWS Resource Groups is a better alternative to the AWS console. This service enables you to create a custom group of your resources, based on specific criteria such as tags or the resources in an AWS CloudFormation stack. By organizing and consolidating information in this way, you can easily track the resources used by individuals or application teams.

It’s worth noting that the service doesn’t support all AWS services (AWS services that work with AWS Resource Groups), as it was not specifically designed for resource discovery. Rather, the service is intended to group resources together based on predetermined tags or CF stack. Therefore, it may not be the best option if you’re looking for a tool to build your asset inventory.

AWS Config

AWS Config is a full-fledged asset inventory. It discovers all your running AWS resources and their configuration history as well as the resource relationships (e.g: find out if an EBS volume is attached to an EC2 instance associated with a security group).

The service also provides a rules engine that you can use to evaluate the configuration of resources against pre-defined rules or compliance policies. E.g: you can use SQL queries to find resources that are non-compliant AWS resources and export the results to JSON or CSV format for further benchmarks (e.g: CIS AWS Benchmarks, AWS Foundational Security Best Practices, or PCI DSS)

Despite those features, the AWS Config service does come with certain drawbacks, including:

As of the time of writing, AWS Config does not cover all types of resources (A list of supported services can be found here).
The more configuration items generated, the more expensive the service can become (See pricing).
AWS Config is best suited for AWS resources. Therefore, users operating in multi-cloud environments and organizations seeking configuration visibility for SaaS assets may require additional tools.
The service is not enabled by default, so users need to set it up in all regions for all their AWS accounts. For those with a considerable number of AWS accounts, this can result in significant effort.

AWS Cost Explorer and CloudWatch

It is also a good idea to take a look at Cost explorer once in a while and check whether we are charging our account unnecessarily. Billing information cannot provide a complete picture. But you can use the AWS Cost Explorer to slice your AWS cost by both AWS services, regions, and tags (if enabled). This can give you a starting point of where to further explore manually with AWS Config or Resource Explorer.

You can also leverage AWS CloudWatch to identify which resources are generating metrics so no resource goes untracked.

AWS Resource Explorer

AWS Resource Explorer is a service released last year that allows you to explore and discover the resources in your AWS account. It allows you to view, search, and filter the resources across all regions and services in your AWS account. The service is free of charge, making it a great alternative to other resource discovery mechanisms, such as AWS Config.

Resource Explorer was built with cross-region support from the very beginning. However, the list of resource types that can be discovered with Resource Explorer is quite short and does not support searching across multiple accounts inside an organization (It only works on an AWS account scope).

As such, you may want to consider alternative options that are more user-friendly and offer a more intuitive way to manage your resources on AWS.

Komiser

Komiser is an open-source cloud-agnostic asset inventory. It integrates with multiple cloud providers, builds a cloud asset inventory, and helps you break down your cost at the resource level.

Komiser comes with a resource inventory feature where you can have an active resource inventory of all your cloud resources along with relevant information such as source account, region, cost, and the tags that are applied to it. You can analyze cloud resource utilization and costs based on specific criteria, such as teams, applications, or cost centers. This approach enables the creation of custom views for engineering, finance, and product teams and promotes accountability for cloud expenses.

As you’re moving to a multi-cloud model, you would need a single place where you can manage all your cloud resources. By integrating with several cloud service providers (currently supporting AWS, Azure, Oracle, DigitalOcean, Civo, Tencent Linode, Kubernetes, and Scaleway), Komiser can swiftly generate your cloud asset inventory. This allows you to utilize its powerful filter system to uncover idle resources and wasted costs across all your cloud accounts and regions. Consequently, supported resources have nowhere to hide, and there is no way they will slip under the radar. As soon as the resource inventory is fetched, all regions will show exactly what they are holding. The resources come to you in a sense, so there’s no more tab switching or console hoping to make sure you didn’t miss anything.

Having an asset inventory of your AWS resources is crucial to uncover optimization opportunities and answering questions about your infrastructure. AWS has some good services but as the number of resources increases and you shift toward multi-cloud you might want to check out something like Komiser that does it all.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

Drift management in cloud infrastructure

LABOUARDY Mohamed — Wed, 22 Feb 2023 17:15:31 +0000

Over the past few years, the number of infrastructure services has grown, and more applications are being released to production on a daily basis while infrastructure needs to be able to be spun up, scaled, and taken down frequently. The adoption of CI/CD and DevOps practices emphasizes the importance of having similar runtime environments. Without an Infrastructure as Code (IaC) practice in place, it becomes increasingly difficult to manage the scale of today’s average infrastructure environment.

IaC safeguards the entire process of cloud provisioning and ensures consistency across different environments by codifying and documenting configuration specifications. IaC tools like Terraform helped the dev and ops teams align as they both use the same description of application deployment. In an ideal world, you want everything to be managed by your IaC stack but expectations do not always line up with reality, resources are still being provisioned manually or through the cloud provider’s consoles, causing infrastructure drift and a growing number of untracked assets.

Understanding the resources that are not managed by IaC in the cloud is a challenge and finding whether they remain in the same configuration defined in the code is yet another task. This blog post will explore the various tools available for detecting and managing infrastructure drift.

What’s Infrastructure Drift

Infrastructure drift occurs when the configuration of the infrastructure deviates from its intended or documented state. This deviation can be attributed to several factors such as human error, lack of automation, manual intervention, applications making unwanted changes, changes applied to some environments but not propagated to others, and so on, leading to inconsistencies in the infrastructure. Additionally, CI/CD workflows can result in failed pipelines, causing the infrastructure state to become corrupted, leading to orphaned resources and drift.

One major cause of infrastructure drift is the creation of resources outside of the established IaC tools such as Terraform, CloudFormation, and Pulumi. When this happens, the infrastructure state is not adequately described or persisted, and the changes made to the infrastructure go unnoticed. This opens the door to security vulnerabilities, wasted costs, and compliance issues.

In some cases, there may be production incidents or emergencies that require quick action, and manual adjustments to the infrastructure via web consoles may be necessary to achieve a better state as soon as possible (and keep the customers satisfied). However, this becomes a problem when those changes are not backported to Terraform which often stems from poor education on best IaC practices, loose access permissions, and a lack of proper communication regarding the infrastructure management process.

Why it’s bad

Infrastructure drift can have a significant impact on the reliability, security, and cost-effectiveness of the infrastructure. One major issue caused by infrastructure drift is the wastage of cloud resources, which can lead to increased costs. Drifting can result in the creation of duplicate resources or the failure to delete unused ones, leading to an unoptimized cloud environment.

Furthermore, infrastructure drift can pose a significant threat to the security of the infrastructure. Inconsistent configurations can make the infrastructure vulnerable to security breaches and data leaks. Such inconsistencies can lead to essential resources unintentionally being made publicly accessible, and unsecured resources may go unnoticed. However, if changes to infrastructure were made through IaC, it would be possible to set up compliance policies and security controls, preventing or mitigating issues such as an S3 bucket being accessible to the public and making sure all resources are properly tagged.

Infrastructure fragmentation is another problem that can arise from infrastructure drift. As the infrastructure becomes more complex, it becomes more difficult to track all resources and changes. This can lead to situations where development teams are unaware of production environment changes, which can cause applications to crash and deployment projects to fail unexpectedly. Moreover, when the IaC tool does not cover the entire infrastructure, it can cause discrepancies between the different environments, leading to inconsistent behavior. This inconsistency can be particularly problematic between the development, staging, and production environments.

Without a single, shared source of truth, intentional infrastructure changes to remediate incidents could be reverted or temporary changes left unnoticed, wasting thousands of dollars in monthly costs due to unused resources.

Cloud workloads undergo frequent changes as more workloads and services are deployed to the infrastructure, resulting in more developers and authenticated services interacting with the infrastructure across various cloud environments and providers. Drift is inevitable, just like incidents, and is a part of the infrastructure’s life cycle. Therefore, it’s crucial to be able to easily and quickly detect and possibly revert drift.

Drift Management

Preventing and resolving infrastructure drift is crucial to maintain the stability and security of the infrastructure. Increasing the adoption of IaC is one of the most effective ways to prevent infrastructure drift. Teams should ensure that a greater percentage of the infrastructure is managed by IaC and leverage code versioning, code reviews, static analysis, automated tests, and so on.

When resources are created using IaC tools, drift can be detected and resolved promptly. For instance, running a command like “terraform plan” can reveal any drift in resources described in the Terraform files.

In the screenshot above, we can see that the EC2 instance owner has changed outside of Terraform which is drift.

CloudFormation has a built-in drift detection feature that can be used either via the AWS Console or via the AWS CLI command.

Regular testing and monitoring are also critical to detect and resolve any issues that may arise due to infrastructure drift. Open-source tools like driftctl, terrascan, and cloud custodian can also be leveraged to detect all changes outside of regular IaC workflow and ensure prompt remediation.

In addition to tracking infrastructure changes, it is crucial to track who is provisioning what, where, and how often. This is especially important since it can be challenging to track those changes across multiple cloud providers and accounts, and manually checking provisioned resources can be time-consuming. Tools like Komiser can be used to build a queryable asset inventory and get a clear picture of the cloud infrastructure. Komiser can detect the drift of managed resources and unmanaged resources in multi-cloud environments, which can be brought under control to maintain consistency and prevent security risks.

After loading the cloud assets into the Komiser dashboard, teams can use filters and views to query the inventory and identify any unmanaged resources. This feature enables you to efficiently manage your cloud infrastructure and ensure that all resources are tracked and appropriately accounted for through your IaC workflows.

Cloud Resource Coverage

In addition to the previous points, it is important to regularly schedule drift detection checks to identify any changes that may have occurred. For instance, an hourly check may be appropriate for detecting any changes in IAM roles, while a daily check may suffice for less critical cloud services. Additionally, to minimize the possibility of infrastructure drift due to manual changes, it is recommended to follow the Least Privilege Principle and restrict permissions to cloud practitioners only for necessary tasks. This approach reduces the number of individuals who can make manual changes to the infrastructure.

In summary, preventing infrastructure drift requires a proactive approach, and a combination of practices and tools can be leveraged to achieve this goal. By increasing IaC adoption, regularly testing and monitoring the infrastructure, and leveraging tools like driftctl and Komiser, teams can detect and resolve drift promptly, maintain consistency, and prevent security risks and bill shocks.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

How to practice FinOps with Komiser

LABOUARDY Mohamed — Mon, 20 Feb 2023 16:49:28 +0000

Financial Operations (FinOps) is a critical aspect of cloud computing that helps organizations to manage their cloud resources effectively and efficiently. With the increasing popularity of cloud computing, the importance of FinOps has only increased (94% of enterprises overspend in the cloud), as organizations look to reduce their cloud spend and make the most of their investments in cloud infrastructure.

Despite the increasing concerns around cloud costs, there has yet to be a single tool that comprehensively manages and helps to remediate excessive cloud expenses. As a result, teams continue to depend on a mix of native cloud provider tools, third-party platforms, and Google Sheets.

According to the State of FinOps 2022 report, teams are still struggling with the following challenges:

FinOps is a cultural discipline that involves collaboration among finance, engineering, product, and management. The open-source model can facilitate this collaboration and cover the long tail of cloud providers and services. That’s where Komiser comes in as a way to address cloud cost management concerns in today’s multi-cloud environments. It offers insight into cloud resource consumption and expenses, making it a valuable tool for organizations practicing FinOps.

In this post, we’ll explore how to use Komiser to empower engineers to optimize their cloud spend while following the FinOps principles.

Cost Allocation

Cost allocation is a crucial component of FinOps. Komiser enables cost allocation to individual projects, teams, or departments, which simplifies the tracking of resource usage and its associated expenses. This information can be used to set budgets, track costs, and identify areas where costs can be optimized. For example, if you see that one team is consistently using more EC2 instances than others, you can work with that team to identify opportunities for optimization.

By utilizing a tagging strategy, you can analyze cloud resource utilization and costs based on specific criteria, such as teams, applications, or cost centers. This approach enables the creation of personalized views that promote accountability for cloud expenses.

This is a great way to show your teams what they’re spending and why and see the impact of their actions on the monthly bill.

Multi-cloud Cost Reporting

As you’re moving to a multi-cloud model, you would need a single place where you can manage all your cloud resources. By integrating with several cloud service providers, Komiser can swiftly generate your cloud asset inventory. This allows you to utilize its powerful filter system to uncover idle resources and wasted costs across all your cloud accounts and regions.

Komiser takes cloud cost management to the next level. Firstly, it gives you visibility into the cloud unit economics that are relevant to you, rather than focusing on a specific cloud vendor. Additionally, it enables you to tag resources across multiple providers and regions using a single interface, which can uncover dormant resources and reduce unnecessary costs.

Komiser provides multi-cloud platform support, including AWS, DigitalOcean, OCI, Tencent, Linode, and Civo, with GCP and Azure support to be added soon. It also supports containerization solutions like Kubernetes.

Uncovering Idle Resources

Komiser provides detailed information on resource utilization, including information on the types of resources being used, the number of resources being used, and the costs associated with each resource. This information can be used to identify opportunities for cost optimization, such as underutilized resources or resources that can be scaled down to reduce costs.

By assigning human-readable labels to cloud resources, teams can use tagging to increase visibility and make smarter budget allocations. With Komiser’s bulk tagging feature, tags can be efficiently applied to a group of resources provisioned in various providers and regions without leaving the Komiser dashboard.

With resources being accurately identified by tags, you can gain a comprehensive understanding of your cloud costs, pinpoint resources that are either redundant or aren’t being used, and identify potential opportunities to save money.

Shared Costs Tracking

A key aspect of FinOps is that every team member is responsible for their cloud usage. The calculation of the total cost of ownership requires transparency and accuracy, but unallocated shared costs obstruct these factors. If shared costs are not properly divided, engineers and product managers do not have a complete understanding of the actual cost of their apps. To improve visibility into shared resources such as databases, logging, k8s, enterprise support, etc., developers can categorize shared resources and allocate budget, as well as create custom views that separate these shared resources from the rest of the team’s views.

Cost Trends Analysis

It’s important to keep an eye on cost trends over time. Komiser provides information on cost trends, including total costs, cost per resource, per team, or tags. This information can be used to identify cost spikes or patterns in resource usage that may indicate an opportunity for optimization. For example, if you see a sudden increase in costs for the frontend team, it may be a good time to review your CDN utilization and see if there are any opportunities to reduce costs.

Komiser cost explorers go beyond native tools like AWS Cost Explorer to provide full-funnel cost visibility across your cloud environment.

Budget Monitoring

In order to stay on top of costs and ensure that resources are being used effectively, it’s important to set alerts in Komiser. Slack alerts can be set to notify you when costs reach a certain threshold, or when resources are being used more than expected. This can help you to catch potential cost optimization opportunities early before they become larger problems.

We need your help

Although Komiser can serve as a good starting point for enabling FinOps within your organization, there is still room for improvement in terms of features. We’re collaborating with the open-source community and cloud leaders to work on the following enhancements:

Automatic resource scaling, right-sizing of instances, RI coverage, and other cost optimization recommendations.
Forecasting based on past cloud usage to anticipate future demand and identify areas for investment.
Additional support for cloud providers and SaaS platforms.
Container cost reporting through OpenCost.

Join our Discord community or visit the Komiser repository to find a good first issue and help us create the future for DevOps where the cloud is transparent and collaborative.

Conclusion

Developing a FinOps culture takes time, but it is essential for creating a sustainable business model. With the right strategy and tools, such as Komiser, you can automate cloud cost management, address complex edge cases, and set higher performance goals. By building a cloud asset inventory, tracking usage and costs, and setting custom alerts, organizations can optimize their cloud infrastructure and make the most of their investments.

Whether you’re just starting out with cloud computing or are an experienced user, Komiser is a valuable tool for practicing FinOps and optimizing cloud spend.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

Manage Kubernetes objects all in one place with Komiser

Jake Page — Wed, 15 Feb 2023 14:25:00 +0000

The cloud observability problem

Managing and optimizing cloud resources is crucial to maximizing their effectiveness, and the right tools can make all the difference when trying to avoid vendor lock-in and tailoring solutions specifically to you. However, many cloud providers struggle to offer a unified and straightforward management platform that works across different cloud environments.

Komiser is an open-source cloud cost optimization tool that can help with this challenge. It provides insights into the costs associated with different regions, managed services, and individual resources, making it a valuable addition to any cloud environment. But not only that, Komiser's Kubernetes integration offers even greater visibility into the Kubernetes clusters running on compute instances, allowing users to create custom views that dynamically update to reflect the current state of their microservice resources.

With Komiser, you can easily gain a complete picture of your cloud cost and usage by leveraging one easy-to-use tool. Komiser offers cloud-agnostic transparency for cost and resource utilization, making it a valuable addition to any cloud environment, regardless of the cloud provider.

Why Kubernetes?

Kubernetes is the present and future of container orchestration. It's a platform that allows you to automate the deployment, scaling, and management of your containerized applications. With Kubernetes, you can build, test, and deploy your applications faster and more reliably. No more manual updates or tedious scripting, Kubernetes does it all for you.

The benefits of Kubernetes are numerous and undeniable. For starters, it increases efficiency and productivity. By automating tasks, developers can focus on writing code and building great products. Kubernetes also provides a level of consistency and reliability that's unparalleled. No more worrying about different environments causing bugs or crashes, Kubernetes ensures your application runs the same way, every time. And let's not forget about scalability. Kubernetes makes it easy to scale your applications up or down, on-demand. So, whether you're running a small blog or a massive enterprise application, Kubernetes has got you covered.

Kubernetes configuration

Komiser now integrates with Kubernetes and takes cloud management to the next level. With this integration, you can now view your Kubernetes objects in the Komiser dashboard, providing complete transparency into your cluster. This gives you the ability to bundle your objects (Deployments, Ingress, Pods, PersistentVolumes, PersistentVolumeClaims, ServiceAccounts, and Services) into custom views, alongside other cloud resources, making it easier to understand and manage your cloud spending. By having all of your resources in one place, you can make informed decisions about how to optimize your costs and ensure that your cluster is running as efficiently as possible. The integration of Komiser with Kubernetes streamlines cloud management and empowers organizations to effectively manage their cloud costs and resources.

Adding kubeconfig to the configuration file

In this example, I’ll be integrating Komiser with three EKS clusters I have deployed to my AWS account.

Easily integrate your Kubernetes clusters by adding a block to the config.toml file as follows:

Above we are integrating three separate Kubernetes accounts named “my-demo-cluster”, “my-demo-staging-cluster” and “my-demo-production-cluster”. Add the path to each individual kubeconfig file and you are all set.

At present, the sole means of cluster integration offered by Komiser's Kubernetes support (v1) is by directly inputting each individual kubeconfig containing the cluster credentials. In the future, we will add support for adding all clusters to the same kubeconfig file and you will be able to load the file once and subsequently choose the cluster profile in each configuration block.

You can locate your kubeconfig file at ~/.kube/config on your local machine.

View of my Kubernetes resources in the Komiser dashboard, which I saved in a Kubernetes resources custom view:

AWS Configuration

To take it a step further, we can update the configuration file to include the AWS account where the Kubernetes clusters we're tracking are located. This enables us to track and manage not only the Kubernetes objects but also the general AWS resources on which the clusters run.

By running this configuration, we can see that the resource inventory page now fetches additional AWS resources, providing us with a more complete picture of our cloud environment. With this enhanced visibility, we can create even more comprehensive custom views and gain greater transparency into our cloud resources.

Resource inventory view

Self-hosting Komiser in a Kubernetes cluster

If you want to self-host Komiser in a Kubernetes cluster, you can use the official Komiser Helm chart to deploy it to any type of Kubernetes cluster. Although the documentation focuses on deploying to an AWS EKS cluster, you can use it to deploy to any other Kubernetes cluster as well.

Tutorial Video

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.

Why we made Komiser open source

LABOUARDY Mohamed — Thu, 09 Feb 2023 13:06:05 +0000

Lost in the clouds

The rise of cloud computing has brought about a number of benefits for businesses of all sizes. From increased speed and flexibility to cost savings and improved security, the cloud has proven to be a valuable asset for many organizations. While the advancement of cloud paradigms like CaaS and FaaS with the integration of Kubernetes and Serverless has made it easier to scale applications without worrying about the underlying infrastructure or vendor lock-in. The widespread use of CI/CD practices and Infrastructure as Code tools has made it much easier to generate and replicate cloud resources across various providers and regions.

Despite the benefits, the cloud computing landscape can be complex and confusing. With the vast array of options available, including multiple pricing models, instance types, storage solutions, and more, along with the widespread adoption of Kubernetes and Serverless, it can be difficult to select the appropriate resources for specific applications and precisely calculate their expenses. Furthermore, the fragmentation of cloud infrastructure across multiple cloud accounts, regions, and providers also leads to a lack of clarity and security, with the existence of dormant, untracked, and idle resources exacerbating these problems.

This is not a new issue, it has been a pain point for several years.

There’s still pain left

A few years ago, while serving as the Head of DevOps at a deep tech startup, I faced the challenge of managing a multi-cloud infrastructure to handle billions of data points daily. Our cloud expenses were rapidly increasing and reached thousands of dollars a day just on AWS, with resources scattered across multiple regions and accounts. Our developers utilized various methods to create cloud resources, such as Cloud Consoles, Dockerfiles, Helm Charts, Jenkinsfiles, Terraform templates, etc., causing infrastructure drift and a growing number of assets that were hard to keep track of. It was difficult to gain a comprehensive understanding of all assets without spending hours searching through multiple AWS Console tabs and layers of hierarchy to answer questions like “How many EC2 instances are running in our Frankfurt region?”

The growth in infrastructure complexity has led to an increase in the number of cloud services used, including compute, storage, and network, among others, each with its own unique pricing structure. This complexity makes it challenging to estimate cloud costs using traditional methods such as spreadsheets. The pressure to deliver results, coupled with the lack of visibility into cloud costs, put DevOps engineers, SREs, and developers at risk of facing backlash from management when unexpected “bill shock” occurs and reaches the CFO.

To address the challenges mentioned, I started by creating a comprehensive view of our cloud resources and their associated costs. Unable to find an existing tool to achieve this, I developed a basic dashboard called Komiser that scans all AWS regions for cloud resources and presents the information in a clear and easy-to-understand format. This gave us complete visibility into all our resources from a single location, making it easier to manage them and avoid costly mistakes. Although the tool was basic, it saved us countless hours of sifting through complex billing statements and reduced the need for frequent context switching.

Once we got the full visibility of our resources as well as the breakdown of their cost and location. We started tagging resources of the cloud environments, giving developers much more visibility and empowering them to take accountability for their cloud spending.

We end up saving thousands of dollars not by implementing an AI-driven cost-saving model but simply by having visibility and taking control of our infrastructure. In addition, our comprehensive inventory knowledge ensured that all assets were properly secured and adhered to the latest security configurations and best practices. This involved monitoring for vulnerabilities, establishing appropriate access controls, and implementing effective network security policies. This gave us a deeper understanding of the entire potential attack surface.

The tool was open-sourced and become a cloud-agnostic with the support of major cloud providers. Upon release, it gained popularity and my colleague Cyril and I noticed that many organizations shared similar challenges, particularly regarding limited visibility into their infrastructure and related tools. To address this challenge, we launched Tailwarden, an open-core company founded on the principles of Komiser and built on an open-source model. Our aim is to empower developers by improving transparency and collaboration in the cloud. Our mission is to put control of the cloud into the hands of developers by tackling one of the most pressing issues in the space.

Why Open-source

We have made Komiser an open-source project due to the significant shift towards open-source software and the numerous advantages it offers in terms of quality, security, and innovation. As Marc Andreessen famously said, “Software is eating the world” and Open Source is now eating software. With the growing number of cloud providers, services, and cloud-native tools, it is challenging to keep up with the wide range of platforms used by developers using a closed-source model. For example, just from re:Invent 2022, AWS introduced 119 new services and features.

Open source is the best approach to tackle the complexity of cloud providers and give developers control over their cloud infrastructure. By opening up Komiser to the open-source community, we gain access to the collective knowledge and expertise of developers from around the world. This enables us to tackle the complexity of supporting a vast array of services and cloud providers.

OSS is built on the foundation of transparency. This aligns with our core values and helps us establish trust with our users. The open-source model ensures accountability and promotes continuous improvement, as our code is openly accessible for review and critique. Unlike closed-source software, where technical debts and flaws may be hidden, open-source software demands the highest quality output. While transparency can be challenging at times, as it puts us under a microscope, it ultimately results in the best possible outcome.

We aim to provide a cloud-agnostic platform that empowers the next generation of developers to build applications without worrying about hidden costs, security issues, or orphaned resources. We believe that with the help of our users and contributors, we can make cloud management easier for everyone.

Where we’re today

Today, Komiser has a growing community of users, with over 3000 stars on GitHub, 3 million downloads, and a thriving community of contributors on Discord.

The tool has proven to be a valuable asset for many developers, helping them to better understand their cloud resources and costs, and making it easier to manage their cloud infrastructure.

You can shape the future of DevOps by contributing to the project, or by providing feedback, be it through GitHub, through the #feedback channel on our Discord server, or by testing existing and new features.

The future of DevOps

Companies are moving to the cloud rapidly and building their entire infrastructure on Cloud providers and SaaS tools. If teams don’t have a single place to manage all of this complexity, they can’t see the big picture. Eventually leaving key insights and opportunities untapped.

Komiser is an essential tool for anyone looking to manage their cloud resources effectively. With its cloud-agnostic approach and open-source model, it can help you better understand your cloud costs and make it easier to manage your resources.