Amazon CloudWatch multi data source querying

Dimple Vaghela — Thu, 01 Feb 2024 18:43:26 +0000

Amazon CloudWatch multi data source querying
Embracing a commitment to innovation and user empowerment, AWS CloudWatch continues to evolve with cutting-edge features that redefine the landscape of cloud monitoring and management. In this blog post, we delve into the latest advancements within AWS CloudWatch, showcasing how these enhancements empower users to gain deeper insights, optimize performance, and seamlessly navigate the dynamic world of cloud computing.

Introduction

Gain visibility across your hybrid and multicloud metrics in a single view with Amazon CloudWatch multi data source querying. With this feature, you can consolidate and visualize metrics from sources such as Amazon OpenSearch Service, Amazon Managed Service for Prometheus, Azure Monitor, your own custom data sources, and query those metrics in real time, increasing visibility into your application health and helping you resolve critical events faster.
As we know now CloudWatch can support multi data source querying. The source can be any of the options listed below.

You have the flexibility to query, visualise, and set alarms for all metrics, irrespective of their origin.

In this blog, we will explore the process of obtaining the source data from an S3 bucket and creating a dashboard—a practical use case that highlights the integration and visualization capabilities within the context of the discussed topic.

Step 1 Create bucket

Create bucket name cw-bucket1

Upload the csv formatted file with the first column timestamp example below.

Select the S3 bucket that you have created

In this example the file name is final.csv

Upload the final.csv file to this bucket

Finally it should look likes as below

So all good to go for CloudWatch now

Step 2 Cloudwatch multi source query initial setup

Access the CloudWatch service console by opening it. Navigate to the Metrics section and expand it.

Then, proceed to click on "All Metrics."

You will observe a display resembling the following.
Select Multi source query

Click “Cloudwatch Metrics Insights” pull dropdown menu

Select “Create and manage data sources”

It will look like the screenshot below. You can see the supported data sources.

Click on getting started.

Select S3 as the data source for the example and click next.

Enter data source name project-csv

Acknowledged to the IAM resources which automatically created with CloudFormation.

Click “Create data source” right side lower cornet.

Once you click it will run a cloudformation template to create resources.

Wait until the cloudformation template create the necessary resources.

After completing the CloudFormation template the below screen will be displayed.

Then the last step to point out the bucket and the key file.

Step 3 Point out the source as bucket and key file.

Click on “Query from Cloudwatch Metrics” link

We need to select the bucket and key

After adding the bucket name and key, it will look like below.

The final steps to query

Step 4 Cloudwatch multi source Query

Click on Graph Query

In case required adjust the time range from top

Based on your data the graph will be displayed

You can play with different kinds of graphs which suit the data.

So finally we finished the Cloudwatch group query with source as S3 Bucket.

Step 5 Final steps is clean up

Open CloudFormation console.

Note : Do not forget to check the Lambda function dashboard.

Just delete the CloudFormation template. Wait till the Cloudformation delete is completed.

Open the S3 console.
Empty the bucket cw-bucket1 and delete the bucket.

Lambda function screenshots Additional.

Conclusion

In conclusion, CloudWatch enables seamless querying of S3 functions without the need for additional code. This blog has provided insights into one of the sources for CloudWatch's multi-source query capabilities, focusing specifically on S3 integration. Stay tuned for future blog posts, where we will explore and delve into other data sources, expanding our understanding of CloudWatch's versatile querying capabilities. The overall takeaway emphasises the simplicity and efficiency with which CloudWatch handles multi-source queries, opening avenues for comprehensive monitoring and analysis across various data streams.