How to report CloudWatch metrics without AWS SDK

Maciej Winnicki — Fri, 18 Mar 2022 09:48:06 +0000

Sometimes when writing a Lambda function, we need to send a custom metric to CloudWatch Metrics. This could be a technical metric (e.g., request time to an external service) or a metric about a business process (e.g., number of user logins). The obvious way to do this is to use the putMetricData method from the AWS SDK:

const AWS = require("aws-sdk");
const cloudwatch = new AWS.CloudWatch();

exports.handler = async () => {
  // business logic

  const metric = {
    Namespace: "Service1",
    MetricData: [
      {
        MetricName: "loginAttempts",
        Dimensions: [
          {
            Name: "tenant",
            Value: "client1",
          },
        ],
        Unit: "Count",
        Value: 1,
      },
    ],
  };

  await cloudwatch.putMetricData(metric).promise();

  // more business logic
};

This solution is sufficient if we occasionally send such metrics. The problem starts if we want to send a lot of those metrics, from different places in the function code. Calling the putMetricData method the same way as calling any other method from AWS SDK, increases the runtime of the function, thus increasing its cost. Besides, we can only send 40kB of data in a single call.
At the end of 2019, AWS has enabled metrics reporting without using AWS SDK.

Embedded Metric Format

This can be done by logging the data to stdout in the specific format called Embedded Metric Format. For example:

{
  "_aws": {
    "Timestamp": 1579211886742,
    "CloudWatchMetrics": [
      {
        "Dimensions": [["tenant"]],
        "Metrics": [
          {
            "Name": "loginAttempts",
            "Unit": "Count"
          }
        ],
        "Namespace": "Service1"
      }
    ]
  },
  "loginAttempts": 1,
  "tenant": "client1"
}

To make it easier to create such an object, AWS has provided libraries for Node.js, Python, Java, and .NET. The above example using the AWS SDK can now be written as follows:

const { createMetricsLogger, Unit } = require("aws-embedded-metrics");

exports.handler = async (event, context) => {
  // business logic

  const metrics = createMetricsLogger();
  metrics.setNamespace("Service1");
  metrics.putMetric("loginAttempts", 1, Unit.Count);
  metrics.setDimensions({ tenant: "client1" });
  await metrics.flush();

  // more business logic
};

There are no network calls to AWS involved, so the function call time does not increase. The data is logged to stdout and the rest is handled by CloudWatch, processing it and publishing it as metrics. Additionally, our function does not need the cloudwatch:PutMetricData permission.

The aws-embedded-metrics library also provides a wrapper for Lambda functions that eliminates the need to manually call the flush() method. This allows reporting metrics to be spread throughout the code and sending them to stdout will only happen once at the end of the Lambda execution.

const { metricScope } = require("aws-embedded-metrics");

exports.handler = metricScope((metrics) => async () => {
  // business logic

  metrics.setNamespace("Service2");
  metrics.putMetric("loginAttempts", 1, Unit.Count);
  metrics.setDimensions({ tenant: "client2" });
  metrics.setProperty("RequestId", context.awsRequestId);

  // more business logic
});

Additionally, using the setProperty method, we can add optional parameters that we can later search for in CloudWatch Logs Insights.

In summary, with Embedded Metric Format, we can optimally send a lot of custom metrics to the CloudWatch service without extending the function duration.

Stay on top of your logs 🌩

Introducing Cloudash, a desktop app for monitoring your serverless services performance, invocations, errors and more.

Did a production incident happen last week? Or 20 seconds ago? With Cloudash you can search, filter and browse your serverless logs and metrics effortlessly.

Search for whatever you want, whenever you want. Cloudash comes with built-in filtering capabilities enabling to get to the bottom of your problems faster than ever before.

Get started here.

A better way of deploying Go services with Serverless Framework

Maciej Winnicki — Mon, 25 Nov 2019 17:39:11 +0000

Go is a great language. I really like it. It's super simple, statically typed, with a huge ecosystem. It makes you productive. Do you know what's better than Go? An AWS Lambda function written in Go and deployed with Serverless Framework.

The experience of building Go services with Serverless Framework is not the best, though. Well, that's pretty obvious because the framework is language-agnostic. Let me list some of the issues:

Go is a compiled language, so you cannot simply run serverless deploy or serverless deploy function after each code change because you need to compile it. It requires an intermediate step making the dev experience a bit disturbed.
Developers that are new to Go don't know all the secrets of go build command. They tend to compile Go code without using addition flags causing the output file being larger than possible, which in turn causes the deployment process to take longer.
When you define the function in serverless.yaml, handler property has to point to the compiled file, rather than to the source file. Again, not big of an issue, but it might be tricky for people starting with Go or/and Serverless Framework.

Usually, the solution is to have a Makefile or some Bash script that will automate the compiling and deploying process. But there is a better way.

The Plugin

I really care about the developer experience, feedback loop, and about using well-tailored tools that make our work easier. That's why I would like to introduce you to ⚡️ Serverless Framework Go Plugin!

mthenw / serverless-go-plugin

⚡️ Serverless Framework plugin that compiles Go functions on the fly. Sponsored by https://cloudash.dev

The plugin solves all the aforementioned issues (and more!) and makes the development process seamless. Let see how it works.

Here is the minimal Go service:

service: example-service

plugins:
  - serverless-go-plugin

provider:
  name: aws
  runtime: go1.x

functions:
  example:
    handler: ./functions/example/main.go
    memorySize: 128

No, let's deploy it:

So:

no reference to the compiled file because the plugin compiles the handler on-the-fly (concurrently on all CPU cores)
no need to set package configuration excluding all unnecessary files from the zipped payload. The plugin does that for you.

The plugin also supports serverless deploy function and serverless invoke local!

Go check it out! If you have any questions let me know in GitHub issues or here in the comments.