Forem: Chris Cook

AI SDK Streaming Text from Lambda

Chris Cook — Wed, 08 Oct 2025 12:13:53 +0000

The AI SDK has two primary modes to interact with a language model: generation and streaming.

Generation means the model returns a complete message. That can be a word, a sentence, paragraph, or complete article.
In streaming mode, the model starts to return multiple partial messages - chunks - while it is generating a response. This is how we usually interact with ChatGPT and other chatbots.

The AI SDK docs already provide lots of examples. While its primary focus might be the usage via Next.js deployed on Vercel, it is built in a vendor-agnostic way so it should run on any Node.js platform.

In this post, I'll port the official example Node.js HTTP text streaming to work on Lambda.

Lambda Response Streaming

AWS introduced response streaming for Lambda functions a while ago, but it looks like it didn't really get into mainstream yet. A reason could be that it is still hard to find examples for using it correctly. Every example I found seems to derive from the official announcement Introducing AWS Lambda response streaming.

exports.handler = awslambda.streamifyResponse(
  async (event, responseStream, context) => {
    responseStream.setContentType("text/plain");
    responseStream.write("Hello, world!");
    responseStream.end();
  }
);

The biggest pain point is the missing type-safety. A few months ago I went through all available docs and examples and extended the official @types/aws-lambda package with types for streaming:

feat(aws-lambda): add response streaming types #72417

zirkelc posted on Apr 07, 2025

Please fill in this template.

[X] Use a meaningful title for the pull request. Include the name of the package modified.
[X] Test the change in your own code. (Compile and run.)
[X] Add or edit tests to reflect the change.
[X] Follow the advice from the readme.
[X] Avoid common mistakes.
[X] Run pnpm test aws-lambda.

Select one of these and delete the others:

If changing an existing definition:

[x] Provide a URL to documentation or source code which provides context for the suggested changes: https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/

View on GitHub

Installing the package will make the global awslambda namespace available, with the streamifyResponse() function and the HttpResponseStream class.

import { StreamifyHandler } from 'aws-lambda';

const streamifyResponseHandler: StreamifyHandler = (event, responseStream, context) => {
  const metadata = {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
      "CustomHeader": "outerspace",
    },
  };
  responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);
  responseStream.setContentType("text/plain");
  responseStream.write("Hello, world!");
  responseStream.end();
};

export const handler = awslambda.streamifyResponse(streamifyResponseHandler);

AI SDK Text Streaming

The official AI SDK example for Node.js uses the pipeTextStreamToResponse function to stream the text back to the client:

import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import { createServer } from 'http';

createServer(async (req, res) => {
  const result = streamText({
    model: openai('gpt-4o'),
    prompt: 'Invent a new holiday and describe its traditions.',
  });

  result.pipeTextStreamToResponse(res);
}).listen(8080);

Looking at the implementation of pipeTextStreamToResponse, it reads the textStream from the streamText() result and writes it to the ServerResponse from node:http.

However, this does not work on Lambda because we cannot return the ServerResponse object; instead, we must write to the output stream responseStream. Since we're dealing with streams, we can use Node.js pipeline() to simply pipe result.textStream to the Lambda response stream.

Here is the Node.js example from above, converted to work on Lambda:

import { bedrock } from '@ai-sdk/amazon-bedrock';
import { streamText } from 'ai';
import { pipeline } from 'node:stream/promises';
import type { APIGatewayProxyEventV2 } from 'aws-lambda';

export const handler = awslambda.streamifyResponse<APIGatewayProxyEventV2>(
  async (event, responseStream) => {
    responseStream.setContentType('text/plain; charset=utf-8');

    const result = streamText({
      model: bedrock('us.anthropic.claude-sonnet-4-5-20250929-v1:0'),
      prompt: 'Invent a new holiday and describe its traditions.',
    });

    await pipeline(result.textStream, responseStream);
  },
);

It's important to set the right content type text/plain via responseStream.setContentType() at the beginning, otherwise your browser or client won't render the incoming text chunks correctly.

Lambda Function URL

Lambda response streaming works only with Function URLs, not with the API Gateway. When you deploy your Lambda function, you must set the InvokeMode to RESPONSE_STREAM:

export class AISDKChatStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const streamFunction = new NodejsFunction(this, 'aisdk-stream', {
      functionName: 'aisdk-stream',
      entry: 'src/functions/stream.ts',
      handler: 'handler',
      runtime: Runtime.NODEJS_22_X,
      timeout: Duration.seconds(60),
      memorySize: 1024,
    });

    const streamFunctionUrl = streamFunction.addFunctionUrl({
      authType: FunctionUrlAuthType.NONE,
      invokeMode: InvokeMode.RESPONSE_STREAM,
      cors: {
        allowedOrigins: Cors.ALL_ORIGINS,
      },
    });
  }
}

Testing in the Browser

When deployed, you can call the Lambda Function URL in the browser and you'll see the output appearing chunk by chunk:

Here's the complete code including the CDK stack to deploy to AWS: github.com/zirkelc/ai-sdk-lambda-streaming

Lambdalet.AI: Reinventing the bookmarklet

Chris Cook — Thu, 10 Jul 2025 10:15:04 +0000

I would like to present Lambdalet.AI, my submission for the AWS Lambda Hackathon.

Lambdalet.AI (Lambda + ~~bookmark~~let) is an AI-powered bookmarking and read-it-later service. It uses a dynamic JavaScript bookmark — a so-called bookmarklet — to send the current page's HTML to an AWS Lambda function. The Lambda function invokes a Large Language Model on Bedrock to extract the page's main content — ignoring headers, footers, and other irrelevant elements — and saves it to a Notion database. The Notion database stores all our bookmarks and allows us to find bookmarks by title, URL, and even their content.

Saving a full page to Notion

What’s a Bookmarklet?

A bookmarklet is a snippet of JavaScript stored in a bookmark; it runs in the context of the page you are viewing by using the javascript: URL scheme. For instance, this bookmarklet makes any website editable:

javascript:(() => {
  document.body.contentEditable = 'true';
  document.designMode = 'on';
  void 0;
})();

Add it as a bookmark, click it on any site (even on this article), and you can freely edit the page.

Bookmarklets were once hugely popular because services such as Instapaper, Evernote, and Pocket could extend browsers without requiring a full extension. The rise of Content Security Policy (CSP) has broken many bookmarklets, even though the spec explicitly states CSP shouldn’t interfere with them. In practice, behavior varies by browser and site, but many pages still allow bookmarklets — or we can often work around these limitations, as we’ll see later.

Why Save Bookmarks to Notion?

Notion is my command center for notes, tasks, and even my blog zirkelc.dev (be sure to give it a star!). Notion databases excel at structured data, and their search is world-class. It always bothered me that browsers save only a title and URL, which often leaves me hunting for an article I know I read but just can’t find again. Capturing the full content solves that.

Try It

You can try Lambdalet.AI right now — no deployment required:

Open the Notion database: Open my shared Notion database in a new tab. New pages will appear here automatically.
Create the bookmarklet: Add a new bookmark and paste this code below as its URL.

Bookmarklet
Pick any website: Like for example, this article. Requests are deduplicated by URL, so multiple clicks on the same page simply update the entry.

Note that if you select any text on the website, only this selection will be saved to Notion
Click the bookmarklet: If the site’s CSP blocks fetch, the code falls back to a form submission in a temporary window.
Check Notion: A new entry appears. Content extraction is asynchronous, so text may take a moment to appear.

How It Works

Let’s follow the request as it travels from your browser to Notion. For more depth, see the repo’s README and the code.

Solution architecture with request flow

On any website you want to save, you can click the bookmarklet in your browser. It will send the current page's HTML, title and URL in a POST request to a REST API Gateway. The bookmarklet is designed to first try a simple fetch call. As this is often blocked by a restrictive Content Security Policy (CSP), the bookmarklet falls back on a form submission which temporarily opens a new window (to avoid navigating way from the current page).

Why POST instead of GET?
A POST request lets the client send the fully rendered HTML — no server-side scraping required. That avoids running a headless browser in Lambda and sidesteps localization issues that would arise if the Lambda IP came from a different region than the user.

The REST API is protected by an API key with usage quotas. Because form submissions can’t set custom headers such as x-api-key, the key is passed as a query-string parameter and verified by a custom Lambda authorizer.

Once past the API Gateway, the request reaches the first Lambda function, whose only job is deduplication. Rather than processing every click immediately, the function writes a message to an SQS FIFO queue keyed by URL. Modern pages easily exceed the 256 KB SQS payload limit, so the raw HTML is uploaded to S3 and the queue message contains only the S3 URI. Multiple clicks on the same URL therefore collapse into a single queued job.

A second Lambda polls the queue. Each message is handled in its own invocation to avoid timeouts while it waits for Bedrock. The function downloads the HTML from S3, then checks whether a Notion page for that URL already exists. If it does, the old page is archived and a new empty page is created. Archiving is cheaper than deleting the page's content block-by-block; the Notion API’s rate limits make large-scale deletes expensive.

The new Notion page initially contains only the title and URL. The raw HTML is first converted to Markdown to reduce token count, then passed to Amazon Bedrock (Claude 3.7 Sonnet) to extract only the main content — no headers, nav bars, cookie banners, or ads.

Prompt

const prompt = `
Here is the content from the URL converted from HTML to markdown:
<url>${url}</url>

<markdown>
${markdown}
</markdown>

Your task is to extract the main content from the given markdown.

Wrap your response in <content> tags.
<content>
[Your markdown content here]
</content>`;

The model's Markdown output is converted into Notion block objects and appended to the page. Finally, the status of the Notion page is updated to indicate the completion of processing.

All Claude models support 200K (~150K words) token context windows, but the practical ceiling is actually the output limit. Only the latest generations, Claude Sonnet 3.7 and 4.0, have raised the limit from 8K to 64K tokens (~48 K words), making such use cases possible.

And that’s it: a click of the bookmarklet becomes a clean, searchable page in Notion.

What’s Next?

I built earlier prototypes in the pre-LLM era, but only recent models, with much larger contexts and outputs, make this practical. Here are a few things I'd like to implement in future iterations.

Richer Metadata

Great bookmarks must be easy to rediscover. Beyond full text, I want tags, cover images, and auto-generated summaries.

Better CSP Handling

The current fallback uses form submission, but some sites — I'm looking at you, GitHub — block even that. I'm exploring additional fallback mechanisms like window.open to open a new tab, though that forces a GET request, so we’d need to fetch HTML server-side — likely with a headless browser.

Refactor with Step Functions

Today the second Lambda idles while waiting for the Bedrock response. Converting the flow to Step Functions would let us use the native Bedrock integration, reducing cost and avoiding timeouts.

Deploy Your Own

Setting up your own Lambdalet.AI instance is easy — full setup instructions are in the repo. Be sure to give it a star and happy bookmarking! 🔖 📑

zirkelc / lambdalet

Lambdalet.AI: Reinventing the bookmarklet.

Lambdalet.AI

Lambdalet.AI (Lambda + ~~bookmark~~let) is an AI-powered bookmarking and read-it-later service. It uses a dynamic javascript bookmark - a so called bookmarklet - to send the current page's HTML to an AWS Lambda function. The Lambda function invokes a Large Language Model on Bedrock to extract the page's main content - ignoring headers, footers and other non-content elements - and saves it to a Notion database. The Notion database stores all our bookmarks and allows us to find bookmarks by title, URL and even their content.

Try It

You can try Lambdalet.AI without deploying anything. Follow these steps:

Open Notion Database: Open the shared Notion database in a new tab. This is where all saved pages will appear.
Create Bookmarklet: Create a new bookmark in your browser using the JavaScript code below as the URL.
JavaScript bookmarklet
```
javascript: (async () => {
```
…

View on GitHub

Flatten Array of Arrays using JSONata for AWS Step Functions

Chris Cook — Tue, 17 Jun 2025 14:38:31 +0000

The Map state of AWS Step Functions returns an array of arrays as a result. For further processing, we often need to flatten this result into a simple one-dimensional flat array. With JSONPath, this can be accomplished using the [*] expression as described in the AWS docs:

"ResultSelector": {
    "flattenArray.$": "$[*][*]"
}

AWS docs for JSONPath

Recently, AWS Step Functions introduced JSONata as an alternative query language to JSONPath. However, since this feature is quite new, you won't find much information on how to accomplish the same with JSONata. At least, this section didn't make it into the AWS docs for JSONata yet.

Note that I'm using states instead of $states in the following examples, because the $ indicates a variable but $states doesn't exist in the context of the JSONata Exerciser.

In JSONata you can use the .* expression to flatten an array of arrays:

JSONata Exerciser

However, you must be aware that this expression returns nothing if the result array is empty:

JSONata Exerciser

In JSONata, you can usually append [] to an expression to cast the result to an array, even if it has only one value:

JSONata Exerciser

However, this trick doesn't work in this case as both [].* and .*[] will return nothing:

JSONata Exerciser

The correct way to handle it is a simple ternary condition using the $exists() function:

JSONata Exerciser

If the expression $exists(states.result.*) evaluates to nothing (falsy value), it will return an empty array as a fallback.

You may ask yourself if there is a shorter version using a short circuit || or null coalescing ?? operator like in JavaScript: states.result.* ?? [].

Unfortunately, this remains an open issue.

TIL `$` captures all positional arguments into a space separated string You can abuse this for a Git alias to avoid typing the commit message in double quotes: Add to .zshrc: alias gc='git add -A && git commit -m "$"' Then run: `gc my commit message`

Chris Cook — Sun, 02 Feb 2025 08:41:49 +0000

TypeScript CLI: Automate Build and Deploy Scripts

Chris Cook — Mon, 27 Jan 2025 11:22:37 +0000

I would like to follow up on my previous post about TypeScript CLIs. Here's how I want to proceed: I plan to implement the build command to build a Vite app and the deploy command to deploy the app to Amazon S3 and AWS CloudFront.

Creating a TypeScript CLI for Your Monorepo

Chris Cook ・ Dec 1 '24

#typescript #javascript #webdev #programming

We will use Listr2 as a task runner to define the steps required to build and deploy the app. We will use execa to run CLI commands for Vite and AWS. Since we're running TypeScript code, we could use the programmatic APIs instead of CLI commands, but let's keep it simple!

#!/usr/bin/env -S pnpm tsx
import chalk from 'chalk';
import { Command } from 'commander';
import { Listr } from 'listr2';
import { $ } from 'execa';

interface Ctx {
  command: 'build' | 'deploy';
}

const tasks = new Listr<Ctx>(
  [
    /**
     * Build tasks
     */
    {
      enabled: (ctx) => ctx.command === 'build' || ctx.command === 'deploy',
      title: 'Build',
      task: (ctx, task): Listr =>
        task.newListr<Ctx>([
          /**
           * Runs `vite build`.
           */
          {
            title: `Run ${chalk.magenta('vite build')}`,
            task: async (ctx, task): Promise<void> => {
              const cmd = $({ all: true })`vite build`;
              cmd.all.pipe(task.stdout());

              await cmd;

              task.output = `Build completed: ${chalk.dim('./dist')}`;
            },
            rendererOptions: { persistentOutput: true },
          },
        ]),
    },
    /**
     * Deploy tasks
     */
    {
      enabled: (ctx) => ctx.command === 'deploy',
      title: 'Deploy',
      task: (ctx, task): Listr =>
        task.newListr<Ctx>([
          /**
           * Runs `aws s3 sync`.
           */
          {
            title: `Run ${chalk.magenta('aws s3 sync')}`,
            task: async (ctx, task): Promise<void> => {
              const build = './dist';
              const bucket = 's3://my-bucket';

              const cmd = $({ all: true })`aws s3 sync ${build} ${bucket} --delete`;
              cmd.all.pipe(task.stdout());

              await cmd;

              task.output = `S3 sync completed: ${chalk.dim(bucket)}`;
            },
            rendererOptions: { persistentOutput: true },
          },
          /**
           * Runs `aws cloudfront create-invalidation`.
           */
          {
            title: `Run ${chalk.magenta('aws cloudfront create-invalidation')}`,
            task: async (ctx, task): Promise<void> => {
              const distributionId = 'E1234567890ABC';

              const cmd = $({ all: true })`aws cloudfront create-invalidation --distribution-id ${distributionId} --paths /* --no-cli-pager`;
              cmd.all.pipe(task.stdout());

              await cmd;

              task.output = `CloudFront invalidation completed: ${chalk.dim(distributionId)}`;
            },
            rendererOptions: { persistentOutput: true },
          },
        ]),
    },
  ],
  {
    rendererOptions: {
      collapseSubtasks: false,
    },
  },
);

const program = new Command()
  .name('monorepo')
  .description('CLI for Monorepo')
  .version('1.0.0');

program
  .command('build')
  .description('Build the monorepo')
  .action(async () => {
    await tasks.run({ command: 'build' });
  });

program
  .command('deploy')
  .description('Deploy the monorepo')
  .action(async () => {
    await tasks.run({ command: 'deploy' });
  });

await program.parseAsync(process.argv);

The tasks are split into build tasks and deploy tasks. Since deploying requires a build step, we use the enabled property to conditionally enable the tasks based on the CLI command build or deploy. Each task executes the corresponding CLI command and pipes its output to the console.

Save this script as cli.ts and run it with pnpm tsx cli:

Unique Symbols: How to Use Symbols for Type Safety

Chris Cook — Wed, 15 Jan 2025 14:09:32 +0000

If you’ve spent some time working with React, you might have stumbled upon React Query’s queryOptions() function. Its implementation looks shockingly simple:

export function queryOptions(options: unknown) {
  return options
}

However, the real magic lies in its overloaded function signatures. So, what’s so special about it?

If you’re unsure what an overloaded function is, you can check out this post: Function Overloading: How to Handle Multiple Function Signatures

Typed Database Queries

Inspired by React Query’s approach, I put together a helper function that might be useful for people working outside of React: a straightforward way to create typed queries, for example, SQL queries.

export declare const queryParamsSymbol: unique symbol;
export declare const queryReturnSymbol: unique symbol;

export type Query<
  TParams extends Record<string, any> = Record<string, any>,
  TReturn extends Record<string, any> | undefined = undefined,
  TStatement extends string = string,
> = {
  statement: TStatement;
  [queryParamsSymbol]: TParams;
  [queryReturnSymbol]: TReturn;
};

export function query<
  TParams extends Record<string, any> = Record<string, any>,
  TReturn extends Record<string, any> | undefined = undefined,
  TStatement extends string = string,
>(statement: TStatement): Query<TParams, TReturn> {
  return { statement: statement } as Query<TParams, TReturn, TStatement>;
}

Similar to queryOptions(), the function itself is pretty dull: it takes a SQL statement, wraps it in an object of type Query, and returns it.

Here’s a quick example of how you’d call it:

const getUserById = query<{ id: number }, { name: string; email: string }>(
  'SELECT name, email FROM users WHERE id = $id',
);

Notice how we pass two types as generic parameters. The first is the required query parameters — in this case, id. The second represents the query’s return type — name and email.

Under the hood, query() embeds those two types into the returned object, stashing them in queryParamsSymbol and queryReturnSymbol. These symbols are declared as unique symbol, which means they only exist in type-space and don’t appear in the transpiled JavaScript.

I use these symbols to temporarily store the parameter and return types and retrieve them whenever I need them.

type InferQueryParams<TQuery> = TQuery extends Query<infer Params, any> ? Params : never;

type UserQueryParams = InferQueryParams<typeof getUserById>;
//        ^? { id: number }

type InferQueryReturn<TQuery> = TQuery extends Query<any, infer Return> ? Return : never;

type UserQueryReturn = InferQueryReturn<typeof getUserById>;
//        ^? { name: string; email: string }

The InferQueryParams and InferQueryReturn are just utility types to confirm that our parameter and return types are being inferred correctly. You might not actually need them, but they’re handy for verifying our approach.

Database Client

Now that we know how to embed parameter and return types in a query object, how do we actually run these queries? Let’s take a look at a simple database client that can execute our typed queries:

class DatabaseClient {
  async execute<
    TParams extends Record<string, any>, 
    TReturn extends Record<string, any>
  >(
    query: Query<TParams, TReturn>,
    params: TParams,
  ): Promise<Array<TReturn>> {
    // execute the query and return the results
    // ...
    return [];
  }
}

const db = new DatabaseClient();

// Return type and parameters are inferred from the query object
const result = await db.execute(getUserById, { id: 1 });
//                                              ^? { id: number }
//      ^? Array<{ name: string; email: string }>
console.log(result);

In this example, we pass our typed getUserById query object to the db.execute() method. Because the Query type contains both the parameter and return-type information, TypeScript automatically infers them. We can easily confirm this by hovering over result and TypeScript will also suggest id as property of our parameters object.

TypeScript Playground

You can find the complete code example in this TypeScript Playground.

Buffer Logs and Flush Automatically on Error with Powertools for Lambda

Chris Cook — Wed, 18 Dec 2024 11:40:32 +0000

The latest release of AWS Powertools for Lambda makes it easier to extend the Logger with custom functionality:

Logger class is now more extensible
You can now overwrite the Logger methods createAndPopulateLogItem, printLog, and processLogItem, which were previously private. This allows you to extend the Logger and add new functionality, e.g., implement your own message buffer.
Release v2.12.0

I'm using this new extensibility to implement a useful feature: log buffering and flushing. The idea is simple: in a production environment, we usually only log important information, such as warnings and errors, because log space can get expensive and it creates noise. However, when an error occurs, we want every possible piece of information: all those debug and info logs scattered throughout a function should be available. But they are not because we set our log level too low.

Buffer and Flush

What if we collect all these debug and info logs internally, and if an important event occurs like an error, we print them to the console? I classify logs into two categories: low-level logs and high-level logs. If our configured log level is WARN, a DEBUG or INFO log would be low-level, and ERROR would be a high-level log.

Now when we print a low-level log, instead of discarding it as it is now, we buffer the log in an internal list. As soon as we have a high-level log, we flush all buffered logs to the console.

Implementation

To add this functionality, we create a new class that extends the Logger class from Powertools and override processLogItem(). This is the central method which is called by the different log methods like logger.debug(). The original implementation prints the log item to the console if it's at the right level. By overriding this method, we can add our special logic of buffering and flushing logs depending on the log level.

import { LogItem, Logger as PowertoolsLogger } from '@aws-lambda-powertools/logger';
import type { LogItemExtraInput, LogItemMessage } from '@aws-lambda-powertools/logger/types';

export class Logger extends PowertoolsLogger {
  #buffer: Record<string, Array<[number, string]>> = {};

  get buffer(): Record<string, Array<[number, string]>> {
    return this.#buffer;
  }

  protected override processLogItem(logLevel: number, input: LogItemMessage, extraInput: LogItemExtraInput): void {
    const xRayTraceId = this['envVarsService'].getXrayTraceId() as string;

    // Flush buffer when log level is higher than the configured log level
    if (logLevel > this.level && xRayTraceId) {
      const buffer = this.#buffer[xRayTraceId] ?? [];

      // Print all log items in the buffer
      if (buffer.length) this.info(`Flushing buffer with ${buffer.length} log items`);

      for (const [bufferLogLevel, bufferLogItem] of buffer) {
        // Create a new LogItem from the stringified log item
        this.printLog(bufferLogLevel, new LogItem(JSON.parse(bufferLogItem)));
      }

      // Clear the buffer after flushing
      // This also removes entries from other X-Ray trace IDs
      this.#buffer = {};
    }

    // Buffer the log item when log level is lower than the configured log level
    if (logLevel < this.level && xRayTraceId) {
      const buffer = this.#buffer[xRayTraceId] ?? [];
      // Add the stringified log item to the buffer
      // Serializing the log item ensures it is not mutated after being added to the buffer
      buffer.push([logLevel, JSON.stringify(this.createAndPopulateLogItem(logLevel, input, extraInput))]);

      // Update the buffer with the new log item
      // This also removes other X-Ray trace IDs from the buffer
      this.#buffer = {
        [xRayTraceId]: buffer,
      };
    }

    // Call the parent method to ensure the log item is processed
    super.processLogItem(logLevel, input, extraInput);
  }
}

You might ask why we use the X-Ray Trace ID here. It's common to instantiate the Logger outside of the handler function. However, because the Lambda execution environment is re-used for potentially multiple invocations, the buffer could contain log items from previous invocations. That's the reason the buffer is implemented as an object rather than a simple array. We use the X-Ray Trace ID as an identifier to only buffer log items from the same invocation.
The buffer is implemented as an object rather than a simple array. When the buffer is flushed, we can simply reset the object and therefore purge items from other invocations.

Test Locally

Let's quickly verify the implementation locally:

// set X-Ray Trace ID manually if running locally
process.env._X_AMZN_TRACE_ID = '1-abcdef12-3456abcdef123456abcdef12';

// log level = WARN
const logger = new Logger({ logLevel: 'WARN' });

logger.debug('debug'); // < log level
logger.info('info');   // < log level
logger.warn('warn');   // = log level
logger.error('error'); // > log level

Here's the output we get:

{"level":"WARN","message":"warn","sampling_rate":0,"service":"service_undefined","timestamp":"2024-12-18T11:01:55.260Z","xray_trace_id":"1-abcdef12-3456abcdef123456abcdef12"}
{"level":"INFO","message":"Flushing buffer with 2 log items","sampling_rate":0,"service":"service_undefined","timestamp":"2024-12-18T11:01:55.261Z","xray_trace_id":"1-abcdef12-3456abcdef123456abcdef12"}
{"level":"DEBUG","message":"debug","sampling_rate":0,"service":"service_undefined","timestamp":"2024-12-18T11:01:55.254Z","xray_trace_id":"1-abcdef12-3456abcdef123456abcdef12"}
{"level":"INFO","message":"info","sampling_rate":0,"service":"service_undefined","timestamp":"2024-12-18T11:01:55.259Z","xray_trace_id":"1-abcdef12-3456abcdef123456abcdef12"}
{"level":"ERROR","message":"error","sampling_rate":0,"service":"service_undefined","timestamp":"2024-12-18T11:01:59.228Z","xray_trace_id":"1-abcdef12-3456abcdef123456abcdef12"}

The warning is the first message because the debug and info logs were buffered. When the error was logged, we flushed the buffered logs (and printed an info) before the error was actually printed.

Request For Comments

My naive implementation has a few caveats. Most importantly, the buffer size is not limited, which means it could cause memory issues if the buffer grows too large. There are a few approaches to mitigate this issue, for example, by implementing the buffer as a sliding window which only keeps the most recent logs or limiting the total buffer size.

Furthermore, the buffered logs are only flushed in controlled cases like on logger.error(), but not on unhandled errors. This behavior could easily be achieved if we make the buffer public and use a middleware like Middy.js. Middy exposes an onError event that we could utilize to flush the buffer.

I have written about this more extensively in this Request For Comment on the official AWS Powertools for Lambda repository.

If you'd like to see this feature become a part of Powertools for Lambda, please share your ideas and feedback there 🙏

How To Remove Dynamic Values From Snapshot With Serializers

Chris Cook — Sat, 14 Dec 2024 09:17:27 +0000

Snapshot tests in Jest and Vitest are powerful tools for detecting unexpected changes in your code's output. However, they easily break when dealing with dynamic values like generated IDs or timestamps that change with each test run. While mocking these values is possible, it can lead to unintended side effects.

Consider this user object which could be returned from an API call or database query:

const user = {
  id: crypto.randomUUID(),
  name: "John Doe",
  createdAt: new Date().toISOString()
};

Every time you run your tests, the id and createdAt values will be different, causing your snapshots to fail.

Basic Implementation

Here's how to create a custom serializer that replaces dynamic values with consistent placeholders:

const property = 'id';
const placeholder = '[ID]';

expect.addSnapshotSerializer({
  test(val) {
    return val && typeof val === 'object' && Object.hasOwn(val, property) && val[property] !== placeholder
  },
  serialize(val, config, indentation, depth, refs, printer) {
    return printer(
      {
        ...(val as Record<string, unknown>),
        [property]: placeholder,
      },
      config,
      indentation,
      depth,
      refs,
    );
  },
});

You can add a custom snapshot serializer with expect.addSnapshotSerializer().
It expects an object with two functions:

test() is used to determine whether this custom serializer should be used. It checks if the value from expect(value) is an object with the property and has not been replaced by the placeholder.
serialize() is only called if test() has returned true. It replaces the property with the placeholder and calls the printer() function to serialize the value into a JSON-like string.

Tests

Now, when you run your tests, you will see that id was replaced with the [ID] placeholder:

interface User {
  id: string;
  name: string;
  createdAt: string;
}

expect.addSnapshotSerializer({ /* ... */ });

test('snapshot', () => {
  const user: User = {
    id: '123e4567-e89b-12d3-a456-426614174000',
    name: 'John Doe',
    createdAt: '2024-03-20T12:00:00Z',
  };

  expect(user).toMatchInlineSnapshot(`
    {
      "id": "[ID]",
      "name": "John Doe",
    }
  `);
});

Making it Reusable

What if we need to handle multiple dynamic properties? Let's create a reusable solution:

export const replaceProperty = (
  property: string,
  placeholder: string,
): SnapshotSerializer => {
  return {
    test(val) {
      return val && typeof val === 'object' && Object.hasOwn(val, property) && val[property] !== placeholder
    },
    serialize(val, config, indentation, depth, refs, printer) {
      return printer(
        {
          ...(val as Record<string, unknown>),
          [property]: placeholder,
        },
        config,
        indentation,
        depth,
        refs,
      );
    },
  };
};

In your tests, you can create multiple serializers for different properties:

expect.addSnapshotSerializer(replaceProperty('id', '[ID]'));
expect.addSnapshotSerializer(replaceProperty('createdAt', '[TIMESTAMP]'));

I use these serializers so frequently that I created the npm package snapshot-serializers to make it easier for everyone.

import { replaceProperty, removeProperty } from 'snapshot-serializers';

type User = {
  id: string;
  name: string;
  createdAt: string;
  password?: string;
};

// Type-safe property replacement
expect.addSnapshotSerializer(
  // TypeScript will only allow "id" | "name" | "createdAt" | "password"
  replaceProperty<User>({
    property: 'id',
    placeholder: '[ID]'
  })
);

// Remove properties entirely
expect.addSnapshotSerializer(
  removeProperty<User>({
    property: 'password'
  })
);

// This would cause a TypeScript error:
expect.addSnapshotSerializer(
  replaceProperty<User>({
    property: 'invalid' // Error: Type '"invalid"' is not assignable...
  })
);

It provides a type-safe API to replace or remove properties in your snapshots. You can provide a generic type parameter like removeProperty<User>() and the function will suggest all possible property names based on the User type. Any other property will cause a TypeScript error.

Creating a TypeScript CLI for Your Monorepo

Chris Cook — Sun, 01 Dec 2024 13:29:13 +0000

I like to create local CLIs for my Monorepo to automate tasks like build and deploy. These tasks often require more than just chaining a few commands in an npm script (like rimraf dist && tsc).

Using commander.js and tsx, we can create executable programs written in TypeScript that run from the command line like any other CLI tool.

#!/usr/bin/env -S pnpm tsx
import { Command } from 'commander';

const program = new Command()
  .name('monorepo')
  .description('CLI for Monorepo')
  .version('1.0.0');

program
  .command('build')
  .description('Build the monorepo')
  .action(async () => {
    console.log('Building...');
    // run your build steps ...
  });

program
  .command('deploy')
  .description('Deploy the monorepo')
  .action(async () => {
    console.log('Deploying...');
    // run your deploy steps ...
  });

await program.parseAsync(process.argv);

Save this script as cli (or any name you prefer) in your project root and make it executable with chmod +x cli. You can then run it directly using ./cli:

$ ./cli
Usage: monorepo [options] [command]

CLI for Monorepo

Options:
  -V, --version   output the version number
  -h, --help      display help for command

Commands:
  build           Build the monorepo
  deploy          Deploy the monorepo
  help [command]  display help for command

The magic that allows you to run this without node, npx, or even a .ts extension is in the first line - the shebang:

#!/usr/bin/env -S pnpm tsx

This shebang tells your shell which program should execute this file. Behind the scenes, it translates your ./cli command into pnpm tsx cli. This works with other package managers too - you can use npm or yarn instead of pnpm.

Middleware for Step Functions: Automatically Store and Load Payloads from Amazon S3

Chris Cook — Tue, 20 Aug 2024 09:01:50 +0000

I would like to introduce middy-store, a new library I built over the last couple of months. I've been pondering this idea for a while, going back to this feature request I opened more than a year ago. middy-store is a middleware for Middy that automatically stores and loads payloads from and to a Store like Amazon S3 or potentially other services.

Motivation

AWS services have certain limits that one must be aware of. For example, AWS Lambda has a payload limit of 6MB for synchronous invocations and 256KB for asynchronous invocations. AWS Step Functions allows for a maximum input or output size of 256KB of data as a UTF-8 encoded string. If you exceed this limit when returning data, you will encounter the infamous States.DataLimitExceeded exception.

The usual workaround for this limitation is to check the size of your payload and save it temporarily in persistent storage such as Amazon S3. Then, you return the object URL or ARN for S3. The next Lambda checks if there is a URL or ARN in the input and loads the payload from S3. As one can imagine, this results in a lot of boilerplate code to store and load the payload from and to Amazon S3, which has to be repeated in every Lambda.

This becomes even more cumbersome when you only want to save part of the payload to S3 and leave the rest as is. For example, when working with Step Functions, the payload could contain control flow data for states like Choice or Map, which has to be accessed directly. This means the first Lambda saves a partial payload to S3, and the next Lambda has to load the partial payload from S3 and merge it with the rest of the payload. This requires ensuring that the types are consistent across multiple functions, which is, of course, very error-prone.

How it works

middy-store is a middleware for Middy. It's attached to a Lambda function and is called twice during a Lambda invocation: before and after the Lambda handler() runs. It receives the input before the handler runs and receives the output from the handler after it has finished.

Let's start at the end with the output after a successful invocation to make it easier to follow: middy-store receives the output (the payload) from the handler() function and checks the size. To calculate the size, it stringifies the payload, if it is an object, and uses Buffer.byteLength() to calculate the UTF-8 encoded string size. If the size is larger than a certain configurable threshold, the payload is stored in a Store like Amazon S3. The reference to the stored payload (e.g., an S3 URL or ARN) is then returned as the output instead of the original output.

Now let's look at the next Lambda function (e.g. in a state machine), which will receive this output as its input. This time we are looking at the input before the handler() is invoked: middy-store receives the input to the handler and searches for a reference to a stored payload. If it finds one, the payload is loaded from the Store and returned as the input to the handler. The handler uses the payload as if it was passed directly to it.

Here's an example to illustrate how middy-store works:

/* ./src/functions/handler1.ts */
export const handler1 = middy()
  .use(
    middyStore({
      stores: [new S3Store({ /* S3 options */ })],
    })
  )
  .handler(async (input) => {
    // Return 1MB of random data as a base64 encoded string as output 
    return randomBytes(1024 * 1024).toString('base64');
  });

/* ./src/functions/handler2.ts */
export const handler2 = middy()
  .use(
    middyStore({
      stores: [new S3Store({ /* S3 options */ })],
    })
  )
  .handler(async (input) => {
    // Print the size of the input
    return console.log(`Size: ${Buffer.from(input, "base64").byteLength / 1024 / 1024} MB`);
  });

/* ./src/workflow.ts */
// First Lambda returns a large output
// It automatically uploads the data to S3 
const output1 = await handler1({});

// Output is a reference to the S3 object: { "@middy-store": "s3://bucket/key"}
console.log(output1); 

// Second Lambda receives the output as input
// It automatically downloads the data from S3
const output2 = await handler2(output1);

What's a Store?

In general, a Store is any service that allows you to store and load arbitrary payloads, like Amazon S3 or other persistent storage systems. Databases like DynamoDB can also act as a Store. The Store receives a payload from the Lambda handler, serializes it (if it's an object), and stores it in persistent storage. When the next Lambda handler needs the payload, the Store loads the payload from the storage, deserializes and returns it.

middy-store interacts with a Store through a StoreInterface interface, which every Store has to implement. The interface defines the functions canStore() and store() to store payloads, and canLoad() and load() to load payloads.

interface StoreInterface<TPayload = unknown, TReference = unknown> {
  name: string;
  canLoad: (args: LoadArgs<unknown>) => boolean;
  load: (args: LoadArgs<TReference | unknown>) => Promise<TPayload>;
  canStore: (args: StoreArgs<TPayload>) => boolean;
  store: (args: StoreArgs<TPayload>) => Promise<TReference>;
}

canStore() serves as a guard to check if the Store can store a given payload. It receives the payload and its byte size and checks if the payload fits within the maximum size limits of the Store. For example, a Store backed by DynamoDB has a maximum item size of 400KB, while an S3 store has effectively no limit on the payload size it can store.
store() receives a payload and stores it in its underlying storage system. It returns a reference to the payload, which is a unique identifier to identify the stored payload within the underlying service. For example, the Amazon S3 Store uses an S3 URI in the format s3://<bucket>/<key> as a reference, while other Amazon services might use ARNs.
canLoad() acts like a filter to check if the Store can load a certain reference. It receives the reference to a stored payload and checks if it's a valid identifier for the underlying storage system. For example, the Amazon S3 Store checks if the reference is a valid S3 URI, while a DynamoDB Store would check if it's a valid ARN.
load() receives the reference to a stored payload and loads the payload from storage. Depending on the Store, the payload will be deserialized into its original type according to the metadata that was stored alongside it. For example, a payload of type application/json will get parsed back into a JSON object, while a plain string of type text/plain will remain unaltered.

Single and Multiple Stores

Most of the time, you will only need one Store, like Amazon S3, which can effectively store any payload. However, middy-store lets you work with multiple Stores at the same time. This can be useful if you want to store different types of payloads in different Stores. For example, you might want to store large payloads in S3 and small payloads in DynamoDB.

middy-store accepts an Array<StoreInterface> in the options to provide one or more Stores. When middy-store runs before the handler and finds a reference in the input, it will iterate over the Stores and call canLoad() with the reference for each Store. The first Store that returns true will be used to load the payload with load().

On the other hand, when middy-store runs after the handler and the output is larger than the maximum allowed size, it will iterate over the Stores and call canStore() for each Store. The first Store that returns true will be used to store the payload with store().

Therefore, it is important to note that the order of the Stores in the array is important.

References

When a payload is stored in a Store, middy-store will return a reference to the stored payload. The reference is a unique identifier to find the stored payload in the Store. The value of the identifier depends on the Store and its configuration. For example, the Amazon S3 Store will use an S3 URI by default. However, it can also be configured to return other formats like an ARN arn:aws:s3:::<bucket>/<key>, an HTTP endpoint https://<bucket>.s3.us-west-1.amazonaws.com/<key>, or a structured object with the bucket and key.

The output from the handler after middy-store will contain the reference to the stored payload:

/* Output with reference */
{
  "@middy-store": "s3://bucket/key"
}

middy-store embeds the reference from the Store in the output as an object with a key "@middy-store". This allows middy-store to quickly find all references when the next Lambda function is called and load the payloads from the Store before the handler runs. In case you are wondering, middy-store recursively iterates through the input object and searches for the "@middy-store" key. That means the input can contain multiple references, even from different Stores, and middy-store will find and load them.

Selecting a Payload

By default, middy-store will store the entire output of the handler as a payload in the Store. However, you can also select only a part of the output to be stored. This is useful for workflows like AWS Step Functions, where you might need some of the data for control flow, e.g., a Choice state.

middy-store accepts a selector in its storingOptions config. The selector is a string path to the relevant value in the output that should be stored.

Here's an example:

const output = {
  a: {
    b: ['foo', 'bar', 'baz'],
  },
};

export const handler = middy()
  .use(
    middyStore({
      stores: [new S3Store({ /* S3 options */ })],
      storingOptions: {
        selector: '',          /* select the entire output as payload */
        // selector: 'a';      /* selects the payload at the path 'a' */
        // selector: 'a.b';    /* selects the payload at the path 'a.b' */
        // selector: 'a.b[0]'; /* selects the payload at the path 'a.b[0]' */
        // selector: 'a.b[*]'; /* selects the payloads at the paths 'a.b[0]', 'a.b[1]', 'a.b[2]', etc. */
      }
    })
  )
  .handler(async () => output);

await handler({});

The default selector is an empty string (or undefined), which selects the entire output as a payload. In this case, middy-store will return an object with only one property, which is the reference to the stored payload.

/* selector: '' */
{
  "@middy-store": "s3://bucket/key"
}

The selectors a, a.b, or a.b[0] select the value at the path and store only this part in the Store. The reference to the stored payload will be inserted at the path in the output, thereby replacing the original value.

/* selector: 'a' */
{
  a: {
    "@middy-store": "s3://bucket/key"
  }
}
/* selector: 'a.b' */
{
  a: {
    b: {
      "@middy-store": "s3://bucket/key"
    }
  }
}
/* selector: 'a.b[0]' */
{
  a: {
    b: [
      { "@middy-store": "s3://bucket/key" }, 
      'bar', 
      'baz'
    ]
  }
}

A selector ending with [*] like a.b[*] acts like an iterator. It will select the array at a.b and store each element in the array in the Store separately. Each element will be replaced with the reference to the stored payload.

/* selector: 'a.b[*]' */
{
  a: {
    b: [
      { "@middy-store": "s3://bucket/key" }, 
      { "@middy-store": "s3://bucket/key" }, 
      { "@middy-store": "s3://bucket/key" }
    ]
  }
}

Size Limit

middy-store will calculate the size of the entire output returned from the handler. The size is calculated by stringifying the output, if it's not already a string, and calculating the UTF-8 encoded size of the string in bytes. It will then compare this size to the configured minSize in the storingOptions config. If the output size is equal to or greater than the minSize, it will store the output or a part of it in the Store.

export const handler = middy()
  .use(
    middyStore({
      stores: [new S3Store({ /* S3 options */ })],
      storingOptions: {
        minSize: Sizes.STEP_FUNCTIONS,  /* 256KB */
        // minSize: Sizes.LAMBDA_SYNC,  /* 6MB */
        // minSize: Sizes.LAMBDA_ASYNC, /* 256KB */
        // minSize: 1024 * 1024,        /* 1MB */
        // minSize: Sizes.ZERO,         /* 0 */
        // minSize: Sizes.INFINITY,     /* Infinity */
        // minSize: Sizes.kb(512),      /* 512KB */
        // minSize: Sizes.mb(1),        /* 1MB */
      }
    })
  )
  .handler(async () => output);

await handler({});

middy-store provides a Sizes helper with some predefined limits for Lambda and Step Functions. If minSize is not specified, it will use Sizes.STEP_FUNCTIONS with 256KB as the default minimum size. The Sizes.ZERO (equal to the number 0) means that middy-store will always store the payload in a Store, ignoring the actual output size. On the other hand, Sizes.INFINITY (equal to Math.POSITIVE_INFINITY) means that it will never store the payload in a Store.

Stores

Currently, there is only one Store implementation for Amazon S3, but I'm planning to implement a Store backed by DynamoDB and DAX. DynamoDB, with its Time-To-Live (TTL) feature, provides a great option for short-term payloads that only need to exist during the execution of a workflow like Step Functions.

Amazon S3

The middy-store-s3 package provides a store implementation for Amazon S3. It uses the official @aws-sdk/client-s3 package to interact with S3.

import { middyStore } from 'middy-store';
import { S3Store } from 'middy-store-s3';

const handler = middy()
  .use(
    middyStore({
      stores: [
        new S3Store({
          config: { region: "us-east-1" },
          bucket: "bucket",
          key: () => randomUUID(),
          format: "arn",
        }),
      ],
    }),
  )
  .handler(async (input) => {
    return { /* ... */ };
  });

The S3Store only requires a bucket where the payloads are being stored. The key is optional and defaults to randomUUID(). The format configures the style of the reference that is returned after a payload is stored. The supported formats include arn, object, or one of the URL formats from the amazon-s3-url package. It's important to note that S3Store can load any of these formats; the format config only concerns the returned reference. The config is the S3 client configuration and is optional. If not set, the S3 client will resolve the config (credentials, region, etc.) from the environment or file system.

Custom Store

A new Store can be implemented as a class or a plain object, as long as it provides the required functions from the StoreInterface interface.

Here's an example of a Store to store and load payloads as base64 encoded data URLs:

import { StoreInterface, middyStore } from 'middy-store';

const base64Store: StoreInterface<string, string> = {
  name: "base64",
  /* Reference must be a string starting with "data:text/plain;base64," */
  canLoad: ({ reference }) => {
    return (
      typeof reference === "string" &&
      reference.startsWith("data:text/plain;base64,")
    );
  },
  /* Decode base64 string and parse into object */
  load: async ({ reference }) => {
    const base64 = reference.replace("data:text/plain;base64,", "");
    return Buffer.from(base64, "base64").toString();
  },
  /* Payload must be a string or an object */
  canStore: ({ payload }) => {
    return typeof payload === "string" || typeof payload === "object";
  },
  /* Stringify object and encode as base64 string */
  store: async ({ payload }) => {
    const base64 = Buffer.from(JSON.stringify(payload)).toString("base64");
    return `data:text/plain;base64,${base64}`;
  },
};

const handler = middy()
  .use(
    middyStore({
      stores: [base64Store],
      storingOptions: {
        minSize: Sizes.ZERO, /* Always store the data */ 
      }
    }),
  )
  .handler(async (input) => {
    /* Random text with 100 words */ 
    return `Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.`;
  });

const output = await handler(null, context);

/* Prints: { '@middy-store': 'data:text/plain;base64,IkxvcmVtIGlwc3VtIGRvbG9yIHNpdC...' } */
console.log(output);

This example is the perfect way to try middy-store, because it doesn't rely on external resources like an S3 bucket. You will find it in the repository at examples/custom-store and should be able to run it locally.

Contributions and Feedback

I've been tinkering with the API design for a while, and it's definitely not stable yet. I would love to get feedback on the current state as well as suggestions for changes or improvements. If you are eager to contribute to this project, please go ahead and submit feature requests or pull requests.

zirkelc / middy-store

Middleware for Step Functions: Automatically Store and Load Payloads

Middleware `middy-store`

middy-store is a middleware for Lambda that automatically stores and loads payloads from and to a Store like Amazon S3 or potentially other services.

Installation

You will need @middy/core >= v5 to use middy-store Please be aware that the API is not stable yet and might change in the future. To avoid accidental breaking changes, please pin the version of middy-store and its sub-packages in your package.json to an exact version.

npm install --save-exact @middy/core middy-store middy-store-s3

Motivation

The usual workaround for this…

View on GitHub

What's a Valuable Skill You Should Have?

Chris Cook — Tue, 30 Jul 2024 04:00:00 +0000

There are certain skills that I always admire in other people. For example, when someone is fluent in using the command line, knows the right commands, and combines them by piping the output from one as input to another.

However, one valuable skill that I think everyone should have - programmer or not - is the ability to read and write Regular Expressions. You don't have to be perfect at it, but you just need to know that they exist, understand their basic structure, and grasp what the different symbols mean.

Regular expressions come in handy in many situations, be it refactoring a codebase with search and replace, searching through log files, or working with spreadsheets.

What do you think is a valuable skill for someone to have?

Collect Logs and Metrics from non-AWS Server using CloudWatch Agent

Chris Cook — Mon, 29 Jul 2024 07:51:34 +0000

Most AWS services provide logs or metrics to CloudWatch, which you can use to monitor your system's health or identify errors. However, services outside of AWS don't inherently benefit from these features. Less well known (judging by the number of GitHub stars) is that AWS offers CloudWatch Agent, an OS service to collect and export system metrics and logs from on-premise servers running outside of AWS.

aws / amazon-cloudwatch-agent

CloudWatch Agent enables you to collect and export host-level metrics and logs on instances running Linux or Windows server.

Amazon CloudWatch Agent

The Amazon CloudWatch Agent is software developed for the CloudWatch Agent

Overview

The Amazon CloudWatch Agent enables you to do the following:

Collect more system-level metrics from Amazon EC2 instances across operating systems. The metrics can include in-guest metrics, in addition to the metrics for EC2 instances. The additional metrics that can be collected are listed in Metrics Collected by the CloudWatch Agent.
Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment as well as servers not managed by AWS.
Retrieve custom metrics from your applications or services using the StatsD and collectd protocols. StatsD is supported on both Linux servers and servers running Windows Server. collectd is supported only on Linux servers.
Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.
Collect Open Telemetry and AWS X-Ray traces

Amazon CloudWatch Agent uses open-source projects…

View on GitHub

In this short post, I've compiled the most important information to help you quickly set up CloudWatch Agent on a server. All information comes from the official documentation, which I'll link for reference. If you prefer a comprehensive overview before starting, I recommend visiting the official docs.

Create IAM User

If your server isn't already using AWS and doesn't have its own user, you'll need to create one. CloudWatch Agent requires permissions to send logs and metrics to CloudWatch. Follow these steps to create a new user. Then, assign one of the managed policies provided by AWS:

I'm using the CloudWatchAgentAdminPolicy, which is almost identical to the CloudWatchAgentServerPolicy. The main difference is that the CloudWatchAgentAdminPolicy allows creating and updating SSM Parameters, while the CloudWatchAgentServerPolicy can only read from SSM. CloudWatch Agent can store and retrieve its config from SSM. When we later initialize CloudWatch Agent, we can directly push the config to SSM.

Configure AWS CLI

If you haven't installed the AWS CLI on your server, follow the instructions in the install guide. CloudWatch Agent uses the profile AmazonCloudWatchAgent for the AWS CLI, so we must configure it with the credentials.

Run the following command and enter the credentials from the previous step:

sudo aws configure --profile AmazonCloudWatchAgent

If you've configured the AWS CLI previously and can't remember the credentials for the IAM user, you can retrieve them by running aws configure get aws_access_key_id and aws configure get aws_secret_access_key.

Install CloudWatch Agent

CloudWatch Agent is available for multiple operating systems and architectures. You can find the appropriate download link for your system in this table.

Amazon provides RPM and DEB packages for Linux. My server runs on Ubuntu, so I'll use the DEB package. It can be downloaded using wget:

wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

After downloading, change to the directory and start the installation. The command depends on the package:

# DEB package
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

# RPM package
sudo rpm -U ./amazon-cloudwatch-agent.rpm

Create Config with Wizard

Before running CloudWatch Agent on your server, you must create a configuration file. The configuration specifies the metrics and logs that the Agent will collect from the server. You can find the supported metrics in this table. The collected logs are specified as file paths like /var/log/auth.log, /var/log/auth.log*, or even /var/log/**.log.

The wizard will take you through a few questions to create a configuration. If you are uncertain about a question, just go with the default choice. You can always start the wizard again or manually edit the config afterward. If you want to store the config as a parameter in SSM, make sure to answer the question with yes and that your IAM user has the CloudWatchAgentAdminPolicy policy.

There's one question regarding running the Agent as the root user or as a different user. The docs mention that CloudWatch Agent runs as the root user by default, but the default choice in the wizard was to run it as user cwagent. If you decide to run it as a different user, be sure to create this user and configure the AWS CLI for it. I overwrote the default choice and run it as the root user.

To start the wizard, run this command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

If you want to double-check your answers, you can see my answers here.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
================================================================
= Welcome to the Amazon CloudWatch Agent Configuration Manager =
=                                                              =
= CloudWatch Agent allows you to collect metrics and logs from =
= your host and send them to CloudWatch. Additional CloudWatch =
= charges may apply.                                           =
================================================================
On which OS are you planning to use the agent?
1. linux
2. windows
3. darwin
default choice: [1]:

Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [2]:

Please make sure the credentials and region set correctly on your hosts.
Refer to http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
Which user are you planning to run the agent?
1. cwagent
2. root
3. others
default choice: [1]:
2
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:

Which port do you want StatsD daemon to listen to?
default choice: [8125]

What is the collect interval for StatsD daemon?
1. 10s
2. 30s
3. 60s
default choice: [1]:
3
What is the aggregation interval for metrics collected by StatsD daemon?
1. Do not aggregate
2. 10s
3. 30s
4. 60s
default choice: [4]:

Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start
1. yes
2. no
default choice: [1]:

Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:

Do you want to monitor cpu metrics per core?
1. yes
2. no
default choice: [1]:
2
Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:

Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:

Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "metrics": {
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "cpu": {
                                "measurement": [
                                        "cpu_usage_idle"
                                ],
                                "metrics_collection_interval": 60,
                                "totalcpu": true
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "diskio": {
                                "measurement": [
                                        "write_bytes",
                                        "read_bytes",
                                        "writes",
                                        "reads"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "net": {
                                "measurement": [
                                        "bytes_sent",
                                        "bytes_recv",
                                        "packets_sent",
                                        "packets_recv"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 60,
                                "service_address": ":8125"
                        },
                        "swap": {
                                "measurement": [
                                        "swap_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:

Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:

Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:

Log file path:
/var/log/auth.log
Log group name:
default choice: [auth.log]
auth-log
Log group class:
1. STANDARD
2. INFREQUENT_ACCESS
default choice: [1]:

Log stream name:
default choice: [{hostname}]
{hostname}      
Log Group Retention in days
1. -1
2. 1
3. 3
4. 5
5. 7
6. 14
7. 30
8. 60
9. 90
10. 120
11. 150
12. 180
13. 365
14. 400
15. 545
16. 731
17. 1096
18. 1827
19. 2192
20. 2557
21. 2922
22. 3288
23. 3653
default choice: [1]:
7
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Do you want the CloudWatch agent to also retrieve X-ray traces?
1. yes
2. no
default choice: [1]:
2
Existing config JSON identified and copied to:  /opt/aws/amazon-cloudwatch-agent/etc/backup-configs
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/auth.log",
                                                "log_group_class": "STANDARD",
                                                "log_group_name": "auth-log",
                                                "log_stream_name": "{hostname}",
                                                "retention_in_days": 30
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "cpu": {
                                "measurement": [
                                        "cpu_usage_idle"
                                ],
                                "metrics_collection_interval": 60,
                                "totalcpu": true
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "diskio": {
                                "measurement": [
                                        "write_bytes",
                                        "read_bytes",
                                        "writes",
                                        "reads"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "net": {
                                "measurement": [
                                        "bytes_sent",
                                        "bytes_recv",
                                        "packets_sent",
                                        "packets_recv"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 60,
                                "service_address": ":8125"
                        },
                        "swap": {
                                "measurement": [
                                        "swap_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:

What parameter store name do you want to use to store your config? (Use 'AmazonCloudWatch-' prefix if you use our managed AWS policy)
default choice: [AmazonCloudWatch-linux]
AmazonCloudWatch-agent-config
Which region do you want to store the config in the parameter store?
default choice: [eu-west-1]

Which AWS credential should be used to send json config to parameter store?
1. XXXXXX... (From SDK)
2. Other
default choice: [1]:

Successfully put config to parameter store AmazonCloudWatch-agent-config.
Program exits now.

Configure credentials

When you start the CloudWatch Agent, it will run as a systemd service. That means, environment variables such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are not accessible.
By default, CloudWatch Agent will search for the credentials in /root/.aws on Linux. Depending on how you installed the AWS CLI, the credentials might be located somewhere else, e.g. usually in the users' /home/<user>/.aws directory. In this case, you need to specify the location of the credentials file in opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

## Configuration for shared credential.
## Default credential strategy will be used if it is absent here:
##            Instance role is used for EC2 case by default.
##            AmazonCloudWatchAgent profile is used for the on-premises case by default.

[credentials]
#  shared_credential_profile = "{profile_name}"
   shared_credential_file= "/home/<user>/.aws/credentials"

See CloudWatch agent credentials preference and Modify the common configuration for proxy or Region information for more information.

Installing `CollectD`

CloudWatch Agent uses the CollectD service to collect metrics. If CollectD is not installed on your system, the Agent will fail to start. If you are not sure if it's installed, here is how you can check if CollectD is installed and active:

sudo systemctl status collectd

This will show you if the service is active, inactive, or if there are any issues with it. Alternatively, you can also look for CollectD in the running processes:

ps aux | grep collectd

If CollectD is not installed, you can install it from your package manager. On Ubuntu, you can install it using apt:

sudo apt update
sudo apt install collectd

Then, start and enable the CollectD service:

sudo systemctl start collectd
sudo systemctl enable collectd

Finally, verify if CollectD is active and running:

sudo systemctl status collectd

Start CloudWatch Agent

The Agent can be controlled by the script located at /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl. If you run it without any arguments, it will show the help:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl

        usage:  amazon-cloudwatch-agent-ctl -a
                stop|start|status|fetch-config|append-config|remove-config|set-log-level
                [-m ec2|onPremise|onPrem|auto]
                [-c default|all|ssm:<parameter-store-name>|file:<file-path>]
                [-s]
                [-l INFO|DEBUG|WARN|ERROR|OFF]

        e.g.
        1. apply a SSM parameter store config on EC2 instance and restart the agent afterwards:
            amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-Config.json -s
        2. append a local json config file on onPremise host and restart the agent afterwards:
            amazon-cloudwatch-agent-ctl -a append-config -m onPremise -c file:/tmp/config.json -s
        3. query agent status:
            amazon-cloudwatch-agent-ctl -a status

        -a: action
            stop:                                   stop the agent process.
            start:                                  start the agent process.
            status:                                 get the status of the agent process.
            fetch-config:                           apply config for agent, followed by -c. Target config can be based on location (ssm parameter store name, file name), or 'default'.
            append-config:                          append json config with the existing json configs if any, followed by -c. Target config can be based on the location (ssm parameter store name, file name), or 'default'.
            remove-config:                          remove config for agent, followed by -c. Target config can be based on the location (ssm parameter store name, file name), or 'all'.
            set-log-level:                          sets the log level, followed by -l to provide the level in all caps.

        -m: mode
            ec2:                                    indicate this is on ec2 host.
            onPremise, onPrem:                      indicate this is on onPremise host.
            auto:                                   use ec2 metadata to determine the environment, may not be accurate if ec2 metadata is not available for some reason on EC2.

        -c: amazon-cloudwatch-agent configuration
            default:                                default configuration for quick trial.
            ssm:<parameter-store-name>:             ssm parameter store name.
            file:<file-path>:                       file path on the host.
            all:                                    all existing configs. Only apply to remove-config action.

        -s: optionally restart after configuring the agent configuration
            this parameter is used for 'fetch-config', 'append-config', 'remove-config' action only.

        -l: log level to set the agent to INFO, DEBUG, WARN, ERROR, or OFF
            this parameter is used for 'set-log-level' only.

CloudWatch Agent must be started with the config created from the wizard. If the config was saved to SSM, use -c ssm:configuration-parameter-store-name with the name of the parameter (not the ARN). Otherwise, use -c file:configuration-file-path with the path to the config file. To start CloudWatch Agent, run the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c [ssm:configuration-parameter-store-name | file:configuration-file-path]

CloudWatch Agent will fetch the config and validate all settings. If everything is fine, it will enable itself as a system service. You can then verify if the Agent is running with this command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m onPremise -a status

Update Config and Restart

If you need to change the config, you can update the SSM parameter or edit the local config file. You can then restart the Agent with the same command as before:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c [ssm:configuration-parameter-store-name | file:configuration-file-path]

The -s option tells CloudWatch Agent to restart itself after the new config is fetched.

Troubleshooting

If the Agent shows an error regarding the credentials or region, check the common configuration for a valid credentials file:

nano /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

If CloudWatch Agent is not working as expected, take a look at the log files in /opt/aws/amazon-cloudwatch-agent/logs:

ls /opt/aws/amazon-cloudwatch-agent/logs/

If the Agent doesn't even start, check the configuration-validation.log file to see if there's an issue with the configuration:

nano /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log

If the Agent is running but the logs and metrics don't show up in CloudWatch, check the amazon-cloudwatch-agent.log file for errors (E!) and warnings (W!):

nano /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Further information can be found in the troubleshooting section in the docs.

Forem: Chris Cook

AI SDK Streaming Text from Lambda

Lambda Response Streaming

feat(aws-lambda): add response streaming types #72417

AI SDK Text Streaming

Lambda Function URL

Testing in the Browser

Lambdalet.AI: Reinventing the bookmarklet

What’s a Bookmarklet?

Why Save Bookmarks to Notion?

Try It

How It Works

What’s Next?

Richer Metadata

Better CSP Handling

Refactor with Step Functions

Deploy Your Own

zirkelc / lambdalet

Lambdalet.AI: Reinventing the bookmarklet.

Lambdalet.AI

Try It

Flatten Array of Arrays using JSONata for AWS Step Functions

TIL `$*` captures all positional arguments into a space separated string You can abuse this for a Git alias to avoid typing the commit message in double quotes: Add to .zshrc: alias gc='git add -A && git commit -m "$*"' Then run: `gc my commit message`

TypeScript CLI: Automate Build and Deploy Scripts

Creating a TypeScript CLI for Your Monorepo

Chris Cook ・ Dec 1 '24

Unique Symbols: How to Use Symbols for Type Safety

Typed Database Queries

Database Client

TypeScript Playground

Buffer Logs and Flush Automatically on Error with Powertools for Lambda

Buffer and Flush

Implementation

Test Locally

Request For Comments

How To Remove Dynamic Values From Snapshot With Serializers

Basic Implementation

Tests

Making it Reusable

Creating a TypeScript CLI for Your Monorepo

Middleware for Step Functions: Automatically Store and Load Payloads from Amazon S3

Motivation

How it works

What's a Store?

Single and Multiple Stores

References

Selecting a Payload

Size Limit

Stores

Amazon S3

Custom Store

Contributions and Feedback

zirkelc / middy-store

Middleware for Step Functions: Automatically Store and Load Payloads

Middleware middy-store

Installation

Motivation

What's a Valuable Skill You Should Have?

Collect Logs and Metrics from non-AWS Server using CloudWatch Agent

aws / amazon-cloudwatch-agent

CloudWatch Agent enables you to collect and export host-level metrics and logs on instances running Linux or Windows server.

Amazon CloudWatch Agent

Overview

Create IAM User

Configure AWS CLI

Install CloudWatch Agent

Create Config with Wizard

Configure credentials

Installing CollectD

Start CloudWatch Agent

Update Config and Restart

Troubleshooting

TIL `$` captures all positional arguments into a space separated string You can abuse this for a Git alias to avoid typing the commit message in double quotes: Add to .zshrc: alias gc='git add -A && git commit -m "$"' Then run: `gc my commit message`

Middleware `middy-store`

Installing `CollectD`