Forem: Daniel Bayerlein

GraphQL Response Streaming with Amazon API Gateway and GraphQL Yoga

Daniel Bayerlein — Fri, 20 Feb 2026 14:16:55 +0000

Imagine you're building a GraphQL API that returns a list of products where each product requires data from multiple services or expensive calculations. Without response streaming, your users wait until all products are fully processed. With the @stream directive, products are sent to the client as soon as they're ready, improving perceived performance.

Until recently, GraphQL response streaming with AWS Lambda was only possible using Lambda Function URLs. But AWS now supports response streaming with Amazon API Gateway, and graphql-yoga has added support for this feature. This opens up new possibilities for building responsive GraphQL APIs with the full feature set of API Gateway (custom domains, usage plans, API keys, etc.).

What's new?

AWS announced support for response streaming in API Gateway in their blog post Building responsive APIs with Amazon API Gateway response streaming.

Instead of buffering the entire Lambda response, API Gateway now streams chunks of data directly to the client as they become available. This not only improves perceived performance but also increases the response payload limit from 6 MB to 20 MB. Combined with GraphQL's @stream directive, you can send data incrementally to your users.

My code examples are written in TypeScript. I use the AWS Cloud Development Kit (CDK), which allows you to define your cloud infrastructure as code in any of the supported programming languages.

GraphQL Server

To enable response streaming with graphql-yoga, you need three key changes:

1. Add the defer-stream plugin

import { useDeferStream } from "@graphql-yoga/plugin-defer-stream";
import type { APIGatewayProxyEvent, Context } from "aws-lambda";
import { createSchema, createYoga } from "graphql-yoga";
import { pipeline } from "stream/promises";

const yoga = createYoga<{
  event: APIGatewayProxyEvent;
  lambdaContext: Context;
  res: awslambda.ResponseStream;
}>({
  plugins: [useDeferStream()],
  schema: createSchema({
    typeDefs: /* GraphQL */ `
      type Product {
        id: ID!
        name: String!
        price: Float!
      }

      type Query {
        products(limit: Int = 100): [Product!]!
      }
    `,
    resolvers: {
      Query: {
        // ...
      }
    },
  }),
});

The @graphql-yoga/plugin-defer-stream plugin enables the @stream and @defer directives in your schema.

2. Use async generators in your resolvers

{
  Query: {
    products: async function* (_parent, args) {
      // Simulate fetching products from a database
      for (let i = 1; i <= args.limit; i++) {
        // Simulate processing delay
        await new Promise((resolve) => setTimeout(resolve, 100));

        yield {
          id: i.toString(),
          name: `Product ${i}`,
          price: Math.random() * 100,
        };
      }
    },
  },
}

The async function* with yield allows graphql-yoga to stream each product as soon as it's ready.

3. Wrap your handler with streamifyResponse

import type { APIGatewayProxyEvent } from "aws-lambda";
import { pipeline } from "stream/promises";

export const handler = awslambda.streamifyResponse(async function handler(
  event: APIGatewayProxyEvent,
  res,
  lambdaContext,
) {
  const queryString = event.queryStringParameters
    ? "?" +
      new URLSearchParams(
        event.queryStringParameters as Record<string, string>,
      ).toString()
    : "";

  const response = await yoga.fetch(
    `https://${event.requestContext.domainName}${event.path}${queryString}`,
    {
      method: event.httpMethod,
      headers: event.headers as HeadersInit,
      body:
        event.body && event.isBase64Encoded
          ? Buffer.from(event.body, "base64")
          : event.body,
    },
    {
      event,
      lambdaContext,
      res,
    },
  );

  res = awslambda.HttpResponseStream.from(res, {
    statusCode: response.status,
    headers: Object.fromEntries(response.headers.entries()),
  });

  if (response.body) {
    await pipeline(response.body, res);
  }

  res.end();
});

awslambda.streamifyResponse enables Lambda's response streaming mode, and pipeline streams the yoga response body to the Lambda response stream.

Using @stream

Now you can use the @stream directive in your queries:

query {
  products(limit: 20) @stream
}

The response arrives in multipart/mixed format with incremental data:

--graphql
Content-Type: application/json

{"data":{"products":[]},"hasNext":true}
--graphql
Content-Type: application/json

{"incremental":[{"items":[{"id":"1","name":"Product 1","price":42.5}],"path":["products",0]}],"hasNext":true}
--graphql
Content-Type: application/json

{"incremental":[{"items":[{"id":"2","name":"Product 2","price":73.2}],"path":["products",1]}],"hasNext":true}
--graphql--

Each product is sent as soon as it's ready.

API Gateway

The API Gateway setup requires one important configuration: responseTransferMode: aws_apigateway.ResponseTransferMode.STREAM.

import {
  aws_apigateway,
  aws_lambda,
  aws_lambda_nodejs,
  Duration,
  Stack,
  type StackProps,
} from "aws-cdk-lib";

import type { Construct } from "constructs";
import * as path from "node:path";

export class GraphqlStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const graphqlLambda = new aws_lambda_nodejs.NodejsFunction(
      this,
      "GraphQLLambdaFunction",
      {
        entry: path.join(__dirname, "../lambda/graphql.ts"),
        architecture: aws_lambda.Architecture.ARM_64,
        runtime: aws_lambda.Runtime.NODEJS_24_X,
        timeout: Duration.seconds(20),
        memorySize: 256,
        bundling: { minify: true },
      },
    );

    new aws_apigateway.LambdaRestApi(this, "GraphQLLambdaRestApi", {
      handler: graphqlLambda,
      integrationOptions: {
        responseTransferMode: aws_apigateway.ResponseTransferMode.STREAM,
      },
    });
  }
}

One thing to watch out for: Make sure your Lambda timeout is sufficient for long-running queries, and remember that you can't change the status code after streaming has started.

Ready!

Response streaming with the @stream directive is a game-changer for GraphQL APIs that require expensive calculations or data from multiple sources. The combination of Amazon API Gateway, AWS Lambda, and graphql-yoga makes it straightforward to implement.

You can find more information in the graphql-yoga documentation, the GraphQL Defer and Stream Directives RFC, and the Amazon API Gateway streaming documentation.

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Building robust and scalable serverless event tracking and analytics with AWS

Daniel Bayerlein — Fri, 28 Feb 2025 13:28:22 +0000

Sometimes, it becomes necessary to develop a custom analytics tracking solution instead of relying on services like Google Analytics. There are several reasons for this. Firstly, a customized solution allows for precise alignment with data privacy laws and regulations, as well as specific compliance requirements. Additionally, it grants full control over the collected data, which is crucial when dealing with sensitive information.

My code examples are written in TypeScript. I use the AWS Cloud Development Kit (CDK), which allows you to define your cloud infrastructure as code in any of the supported programming languages.

Challenge

There are a few important points to think about when talking about a user tracking solution. One of them is the large amount of data generated by user interactions. The endpoint that receives this event data needs to be able to process thousands of data points within seconds.

The amount of data is also a challenge for the analysis. Therefore, a solution is needed that can process thousands of user events very efficiently and quickly.

Let's take a look at the architecture and the services that are used.

Architecture

In the client application the Amplify Analytics library is used. This provides a ready-to-use implementation that also takes care of batch processing and authentication using Amazon Cognito.

Amplify Analytics uses Amazon Cognito Identity Pools to identify users in the app. Cognito can be used to receive data from authenticated and unauthenticated users in the app.

The Amazon Data Firehose analytics provider allows you to send analytics data to an Amazon Data Firehose stream for reliably storing data.

Amazon Data Firehose sends the streamed data to an Amazon S3 bucket. The data is stored there batched as an S3 object by default.

The Amazon Athena service makes it easy to analyze data in Amazon S3 using standard SQL. Athena uses the AWS Glue Data Catalog to store and retrieve table metadata for the Amazon S3 data. The table metadata lets the Athena query engine know how to find, read, and process the data that you want to query.

Infrastructure

Every user can receive temporary permissions that allows them to write data directly to Data Firehose. This can be done with a Cognito Identity Pool. For more details about authenticated and unauthenticated user please have a look at the Cognito documentation.

This example uses an existing Cognito UserPool and UserPoolClient as a provider for authenticated users.

import {
  IdentityPool,
  UserPoolAuthenticationProvider,
} from '@aws-cdk/aws-cognito-identitypool-alpha';

const identityPool = new IdentityPool(this, 'AnalyticsIdentityPool', {
  authenticationProviders: {
    userPools: [
      new UserPoolAuthenticationProvider({
        userPool,
        userPoolClient,
      }),
    ],
  },
});

Data Firehose requires an Amazon S3 bucket to store the data there.

import { aws_s3 } from 'aws-cdk-lib';

const bucket = new aws_s3.Bucket(this, 'AnalyticsBucket', {
  blockPublicAccess: aws_s3.BlockPublicAccess.BLOCK_ALL,
});

A Firehose Delivery Stream is provided as the recipient for the user events. The S3 bucket is used as the destination for all streaming data.

As the data analysis is to be carried out on a daily basis, a very common structure is used within the S3 bucket:

/{year}/{month}/{day}/

This structure can be configured with the dataOutputPrefix property.

import { aws_kinesisfirehose } from 'aws-cdk-lib';

const firehoseDeliveryStream = new aws_kinesisfirehose.DeliveryStream(
  this,
  'AnalyticsDeliveryStream',
  {
    deliveryStreamName: 'analyticsStream',
    destination: new aws_kinesisfirehose.S3Bucket(bucket, {
      dataOutputPrefix:
        'data/!{partitionKeyFromQuery:year}/!{partitionKeyFromQuery:month}/!{partitionKeyFromQuery:day}/',
      errorOutputPrefix:
        'failures/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/',
    }),
  },
);

firehoseDeliveryStream.grant(
  identityPool.authenticatedRole,
  'firehose:PutRecord',
  'firehose:PutRecordBatch',
);

As not all service properties are yet implemented in the Construct, some properties must be overwritten directly on the delivery stream node.

Enable configuration of the dynamic partitioning mechanism with the ExtendedS3DestinationConfiguration.DynamicPartitioningConfigurationproperty type, which creates targeted datasets from streaming data by partitioning it based on partition keys.

The ExtendedS3DestinationConfiguration.ProcessingConfiguration property type configures an Amazon S3 destination for an Amazon Kinesis Data Firehose delivery stream.

import { aws_kinesisfirehose } from 'aws-cdk-lib';

const firehose = firehoseDeliveryStream.node
  .defaultChild as aws_kinesisfirehose.CfnDeliveryStream;

firehose.addPropertyOverride(
  'ExtendedS3DestinationConfiguration.DynamicPartitioningConfiguration',
  { Enabled: true },
);

firehose.addPropertyOverride(
  'ExtendedS3DestinationConfiguration.ProcessingConfiguration',
  {
    Enabled: true,
    Processors: [
      {
        Type: 'RecordDeAggregation',
        Parameters: [
          {
            ParameterName: 'SubRecordType',
            ParameterValue: 'JSON',
          },
        ],
      },
      {
        Type: 'AppendDelimiterToRecord',
      },
      {
        Type: 'MetadataExtraction',
        Parameters: [
          {
            ParameterName: 'MetadataExtractionQuery',
            ParameterValue:
              '{year:.event_timestamp| strftime("%Y"),month:.event_timestamp| strftime("%m"),day:.event_timestamp| strftime("%d")}',
          },
          {
            ParameterName: 'JsonParsingEngine',
            ParameterValue: 'JQ-1.6',
          },
        ],
      },
    ],
  },
);

⚠️ At the time of writing, the AWS Glue CDK Construct (@aws-cdk/aws-glue-alpha) was still experimental.

Create an AWS Glue database and table. Configure the table with a partition based on event_date and columns for event and event_timestamp.

Customize the table properties to enable and configure partition projection.

import * as aws_glue from '@aws-cdk/aws-glue-alpha';
import type { CfnTable } from 'aws-cdk-lib/aws-glue';

const database = new aws_glue.Database(this, 'AnalyticsDatabase');
const table = new aws_glue.Table(this, 'AnalyticsTable', {
  bucket,
  s3Prefix: 'data/',
  dataFormat: aws_glue.DataFormat.JSON,
  database,
  partitionKeys: [
    {
      name: 'event_date',
      type: aws_glue.Schema.STRING,
    },
  ],
  columns: [
    {
      name: 'event',
      type: aws_glue.Schema.STRING,
    },
    {
      name: 'event_timestamp',
      type: aws_glue.Schema.TIMESTAMP,
    },
  ],
});

const cfnTable = table.node.defaultChild as CfnTable;
cfnTable.addPropertyOverride('TableInput.Parameters', {
  'projection.enabled': 'true',
  'projection.event_date.type': 'date',
  'projection.event_date.format': 'yyyy/MM/dd',
  'projection.event_date.range': '2024/01/01,NOW',
  'projection.event_date.interval.unit': 'DAYS',
  'projection.event_date.interval': '1',
  'storage.location.template':
    analyticsBucket.s3UrlForObject() + '/data/${event_date}/',
});

If you want to store additional data, add a column for each data point.

The infrastructure for the serverless event tracking solution is now in place!

Client

Now let's take a look at the client application. As described above, aws-amplify/analytics provides a ready-to-use implementation for Amazon Kinesis Firehose.

npm install aws-amplify

Ensure you have setup IAM permissions for firehose:PutRecordBatch.

Example IAM policy for Amazon Kinesis Firehose:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "firehose:PutRecordBatch",
    // replace the template fields
    "Resource": "arn:aws:firehose:<Region>:<AccountId>:deliverystream/analyticsStream"
  }]
}

Configure Kinesis Firehose:

import { Amplify } from 'aws-amplify';

Amplify.configure({
  Analytics: {
    KinesisFirehose: {
      // REQUIRED - Amazon Kinesis Firehose service region
      region: 'eu-central-1',

      // OPTIONAL - The buffer size for events in number of items.
      bufferSize: 1000,

      // OPTIONAL - The number of events to be deleted from the buffer when flushed.
      flushSize: 100,

      // OPTIONAL - The interval in milliseconds to perform a buffer check and flush if necessary.
      flushInterval: 5000, // 5s

      // OPTIONAL - The limit for failed recording retries.
      resendLimit: 5
    }
  }
});

You can send a data to a Kinesis Firehose stream with the standard record method. Any data is acceptable and streamName is required:

import { record } from 'aws-amplify/analytics/kinesis-firehose';

record({
  data: {
    event_timestamp: Date.now() / 1000,
    event: 'LOGIN',
    // The data blob to put into the record
  },
  streamName: 'analyticsStream'
});

If you want to store additional data, add it to the data object and remember to add a specific column to the AWS Glue table.

That's it. Now you can track events in your application however you like.

Analysis

Now that the infrastructure and client application are in place, it is time to analyze the data. This can be done using the Amazon Athena service. The analysis can be performed directly in the AWS Console or via the AWS SDK if you want to visualize the results in an application.

AWS Glue

When you look at the AWS Glue service, you can see the metadata of the table.

These columns can now be queried using Amazon Athena.

Amazon Athena

Amazon Athena makes it easy to analyze data in Amazon S3 using standard SQL. For more information, see 📖 SQL reference for Athena

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Run LLMs locally with Ollama on macOS for Developers

Daniel Bayerlein — Wed, 05 Feb 2025 14:37:20 +0000

In this blog post, I will show you how to run LLMs locally on macOS. I also want to provide an AI interface and integrate an AI coding assistant into VS Code.

Ollama

Ollama is an open source tool that allows you to run large language models (LLMs) directly on a local machine.

I use Homebrew for the installation of Ollama. But there are also alternative installation options available, see https://ollama.com/download.

Please run Ollama as a standalone application outside of Docker containers as Docker Desktop does not support GPUs.
Source

Use the following command to install Ollama:

brew install ollama

To run Ollama as a service in the background, execute the following command:

brew services start ollama

Ollama should now be running and accessible at http://localhost:11434/. If you open the URL in your browser, you will see Ollama is running.

You can now download the LLM of your choice. An overview of all available LLMs can be found here.

Let's start with the DeepSeek-R1 LLM. You can download it using the following command:

ollama pull deepseek-r1

Ready to go. Run the model with the following command:

ollama run deepseek-r1

Now ask DeepSeek-R1 something.

AI interface

As an alternative to the command line interface, I would like to use a user interface. So take a look at Open WebUI.

Open WebUI is an extensible, self-hosted AI interface that adapts to your workflow, all while operating entirely offline.

If you are already using Docker, installation is straightforward. Run the following command:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

See documentation for an alternative installation method.

Open WebUI is now available for use on http://localhost:3000/.

AI Coding Assistant in Visual Studio Code

I would like a local AI coding assistant in VS Code as an alternative to GitHub Copilot. There are a number of different extensions, one of which is Continue.

The leading open-source AI code assistant. You can connect any models and any context to create custom autocomplete and chat experiences inside the IDE

You can find the extension for VS Code in the Visual Studio Marketplace.

Continue provides the following features:

Chat to understand and iterate on code in the sidebar
Autocomplete to receive inline code suggestions as you type
Edit to modify code without leaving your current file
Actions to establish shortcuts for common use cases

I want to use Llama 3.1 8B model for chat and Qwen2.5-Coder 1.5B model for autocomplete.

These models can be provided with Ollama using the following command:

ollama pull llama3.1:8b
ollama pull qwen2.5-coder:1.5b

The VS Code extension Continue must now be configured:

{
  "models": [
    {
      "title": "Llama 3.1 8B",
      "model": "llama3.1:8b",
      "provider": "ollama",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder 1.5B",
    "model": "qwen2.5-coder:1.5b",
    "provider": "ollama",
    "apiBase": "http://localhost:11434"
  },
}

provider is always ollama and apiBase is always the path where Ollama can be accessed (http://localhost:11434).

The Continue features are now available in VS Code and can be used.

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Malware Protection for S3 with Amazon GuardDuty

Daniel Bayerlein — Fri, 31 Jan 2025 07:36:09 +0000

Malicious files uploaded to your Amazon S3 bucket? You protect your S3 bucket, right? Right!?

To protect against this, it was usually necessary to setting up secure staging buckets, using anti-virus and anti-malware scanning software, and managing a data pipeline and processing architecture.

In this blog post, I explain how to detect malicious file uploads to your Amazon S3 bucket and how you can take action to isolate or eliminate any malware found.

My code examples are written in TypeScript. I use the AWS Cloud Development Kit (CDK), which allows you to define your cloud infrastructure as code in any of the supported programming languages.

Amazon GuardDuty Malware Protection for S3

AWS released Amazon GuardDuty Malware Protection for S3 at re:Inforce 2024.

It's an advanced security feature that extends the capabilities of Amazon GuardDuty. GuardDuty is an intelligent threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts, workloads, and data. With the addition of Malware Protection for S3, GuardDuty offers comprehensive protection for your S3 buckets.

Amazon GuardDuty Malware Protection uses multiple AWS developed and industry-leading third-party malware scanning engines to provide malware detection.

Code example

For the sake of completeness, I create a customer-managed key and an S3 bucket. If these already exist, you can skip this step.

import { aws_kms, aws_s3 } from 'aws-cdk-lib';

const kmsKey = new aws_kms.Key(this, 'ExampleKmsKey');

const bucket = new aws_s3.Bucket(this, 'ExampleBucket', {
  bucketKeyEnabled: true,
  encryption: aws_s3.BucketEncryption.KMS,
  encryptionKey: kmsKey,
});

To enable Malware Protection for S3 to scan and tag your S3 objects, you can use service roles that have the necessary permissions to perform malware scanning actions on your behalf.

The following permissions are required to perform the malware scan:

Allow Amazon EventBridge actions to create and manage the EventBridge managed rule so that Malware Protection for S3 can listen to your S3 object notifications.
Allow Amazon S3 and EventBridge actions to send notiﬁcation to EventBridge for all events in this bucket.
Allow Amazon S3 actions to access the uploaded S3 object.
Allow KMS key actions to access the object.

First, create the IAM policy. Don't be surprised about the DO-NOT-DELETE-AmazonGuardDutyMalwareProtectionS3 EventBridge rules, these are managed by AWS.

import { aws_iam, aws_kms, aws_s3 } from 'aws-cdk-lib';

...

const policy = new aws_iam.Policy(
  this,
  'GuardDutyMalwareProtectionRolePolicy',
  {
    statements: [
      new aws_iam.PolicyStatement({
        sid: 'AllowManagedRuleToSendS3EventsToGuardDuty',
        effect: aws_iam.Effect.ALLOW,
        actions: [
          'events:PutRule',
          'events:DeleteRule',
          'events:PutTargets',
          'events:RemoveTargets',
        ],
        resources: [
          `arn:aws:events:${this.region}:${this.account}:rule/DO-NOT-DELETE-AmazonGuardDutyMalwareProtectionS3*`,
        ],
        conditions: {
          StringLike: {
            'events:ManagedBy':
              'malware-protection-plan.guardduty.amazonaws.com',
          },
        },
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowGuardDutyToMonitorEventBridgeManagedRule',
        effect: aws_iam.Effect.ALLOW,
        actions: ['events:DescribeRule', 'events:ListTargetsByRule'],
        resources: [
          `arn:aws:events:${this.region}:${this.account}:rule/DO-NOT-DELETE-AmazonGuardDutyMalwareProtectionS3*`,
        ],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowPostScanTag',
        effect: aws_iam.Effect.ALLOW,
        actions: [
          's3:PutObjectTagging',
          's3:GetObjectTagging',
          's3:PutObjectVersionTagging',
          's3:GetObjectVersionTagging',
        ],
        resources: [bucket.arnForObjects('*')],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowEnableS3EventBridgeEvents',
        effect: aws_iam.Effect.ALLOW,
        actions: ['s3:PutBucketNotification', 's3:GetBucketNotification'],
        resources: [bucket.bucketArn],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowPutValidationObject',
        effect: aws_iam.Effect.ALLOW,
        actions: ['s3:PutObject'],
        resources: [
          `${bucket.bucketArn}/malware-protection-resource-validation-object`,
        ],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowCheckBucketOwnership',
        effect: aws_iam.Effect.ALLOW,
        actions: ['s3:ListBucket', 's3:GetBucketLocation'],
        resources: [bucket.bucketArn],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowMalwareScan',
        effect: aws_iam.Effect.ALLOW,
        actions: ['s3:GetObject', 's3:GetObjectVersion'],
        resources: [bucket.arnForObjects('*')],
      }),
      new aws_iam.PolicyStatement({
        sid: 'AllowDecryptForMalwareScan',
        effect: aws_iam.Effect.ALLOW,
        actions: ['kms:GenerateDataKey', 'kms:Decrypt'],
        resources: [kmsKey.keyArn],
        conditions: {
          StringLike: {
            'kms:ViaService': `s3.${this.region}.amazonaws.com`,
          },
        },
      }),
    ],
  },
);

Once the IAM policy is defined, the IAM role can be created and then added to the policy.

import { aws_iam, aws_kms, aws_s3 } from 'aws-cdk-lib';

...

const role = new aws_iam.Role(this, 'GuardDutyMalwareProtectionPassRole', {
  assumedBy: new aws_iam.ServicePrincipal(
    'malware-protection-plan.guardduty.amazonaws.com',
  ),
  description:
    'An iam pass role for guardduty malware protection service to assume',
});

policy.attachToRole(role);

Finally, create a new malware protection plan for the protected resource. The tagging of S3 objects is also enabled. More on this in a moment.

import { aws_guardduty, aws_iam, aws_kms, aws_s3 } from 'aws-cdk-lib';

...

const malwareProtectionPlan = new aws_guardduty.CfnMalwareProtectionPlan(
  this,
  'GuardDutyMalwareProtection',
  {
    protectedResource: {
      s3Bucket: {
        bucketName: bucket.bucketName,
      },
    },
    role: role.roleArn,
    actions: {
      tagging: {
        status: 'ENABLED',
      },
    },
  },
);

Everything looks good - the deployment is running and then suddenly fails 🤯

The request was rejected because provided IAM role does not have the required permissions to validate S3 bucket ownership

Sometimes the AWS CloudFormation stack is attempting to create the CfnMalwareProtectionPlan resource before the IAM role has been fully created and propagated across AWS services. You can fix the problem by defining the order of the dependencies.

malwareProtectionPlan.node.addDependency(policy);

The deployment should now be successful and GuardDuty is ready to use. 🚀

Take action

There are several ways to proceed from here. I would like to briefly present two possible solutions.

Tag-based access control (TBAC)

Because the S3 objects are tagged, a tag-based Access Control (TBAC) resource policy can be used.

As long as an object does not have a tag or the tag does not match GuardDutyMalwareScanStatus:NO_THREATS_FOUND, access to it can be prevented.

For more information, see 📖 Using tag-based access control (TBAC) with Malware Protection for S3.

Copy S3 objects

Another solution would be to copy the S3 objects that have been scanned by Amazon GuardDuty to a different S3 bucket.

An EventBridge rule is configured to listen for events that match a scan result of NO_THREATS_FOUND. A Lambda function is then called to copy the S3 object to another S3 bucket.

For more information, see 📖 Amazon GuardDuty copy S3 object solution overview.

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Using pnpm with the GitLab package registry in GitLab CI

Daniel Bayerlein — Fri, 08 Mar 2024 11:47:03 +0000

In this blog post, I explain how to use pnpm in GitLab CI and how to authenticate with a private GitLab package registry.

Corepack is an experimental tool that helps you manage the versions of your package managers.

package.json

The packageManager field specifies which package manager is expected to be used. The value is set to pnpm with the preferred version 8.15.4.

{
  "packageManager": "pnpm@8.15.4"
}

.gitlab-ci.yml

In order to have pnpm available in every job, it is installed within the before_script. These commands are executed before each job!

before_script:
  - corepack enable
  - corepack prepare --activate

To also use the cache, it is configured as follows:

cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - .pnpm-store
  policy: pull-push

before_script:
  - pnpm config set store-dir .pnpm-store

The following commands are required to access the private package registry:

before_script:
  - pnpm config set @scope:registry https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/
  - pnpm config set -- //${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken ${CI_JOB_TOKEN}

Replace @scope with your package scope. The environment variables CI_SERVER_HOST, CI_PROJECT_ID and CI_JOB_TOKEN are provided by GitLab CI. You do not need to manually create and set a token for the package registry. The CI_JOB_TOKEN is valid as long as the job is running.

Packages can now be downloaded and published.

Complete code example

With this example you can use pnpm in GitLab CI, download packages from the GitLab package registry and also publish them.

.gitlab-ci.yml

image: 'public.ecr.aws/docker/library/node:18-bullseye'

stages:
  - build
  - deploy

cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - .pnpm-store
  policy: pull-push

before_script:
  - corepack enable
  - corepack prepare --activate
  - pnpm config set store-dir .pnpm-store
  - pnpm config set @scope:registry https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/
  - pnpm config set -- //${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken ${CI_JOB_TOKEN}
  - pnpm install

build:
  stage: build
  script:
    - pnpm run build
  artifacts:
    paths:
      - dist/

publish:
  stage: deploy
  needs:
    - job: build
  script:
    - pnpm publish

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Encrypt auto created log groups from AWS Lambda with AWS CDK

Daniel Bayerlein — Wed, 16 Nov 2022 14:10:02 +0000

First of all: Log group data is always encrypted in Amazon CloudWatch Logs. By default, CloudWatch Logs uses server-side encryption for the log data at rest.

However, sometimes it is necessary - for example, because of compliance guidelines - to encrypt the logs with a customer managed key. No problem, you can use AWS Key Management Service (AWS KMS) for this encryption.

But how does this work with automatically created log groups by AWS services, like AWS Lambda? In this blog post, I want to show you what steps are necessary to do this.

Enough introduction, let's get to the code.

My code examples are written in TypeScript. I use the AWS Cloud Development Kit (CDK), which allows you to define your cloud infrastructure as code in any of the supported programming languages.

AWS KMS

I use the AWS KMS Construct to define a new KMS key:

const kmsKey = new aws_kms.Key(this, 'KmsKey');

Now I grant the CloudWatch Logs service encryption and decryption rights for my region using this KMS key:

kmsKey.grantEncryptDecrypt(
  new aws_iam.ServicePrincipal(`logs.${this.region}.amazonaws.com`)
);

This was the necessary configuration for AWS KMS. The key can now be used.

AWS Lambda

AWS Lambda automatically creates a log group if it does not already exist. Unfortunately, the log group cannot really be configured using the AWS Lambda Construct. With one small exception: The logRetention can be specified.

For completeness, I create a Lambda function where the log group is to be encrypted with a KMS key:

const lambdaFn = new aws_lambda_nodejs.NodejsFunction(this, 'LambdaFn', {
  entry: '/path/to/my/file.ts'
});

Now I create a new log group using the AWS LogGroup Construct:

new aws_logs.LogGroup(this, 'LambdaLogGroup', {
  encryptionKey: kmsKey,
  logGroupName: `/aws/lambda/${lambdaFn.functionName}`
});

⚠️ With the encryptionKey property I define which KMS key should be used to encrypt the log group. The important step is the logGroupName property. By default, the log group of the Lambda function is created with the prefix /aws/lambda/, followed by the function name. Naming the log group according to this naming scheme ensures that the Lambda function uses it.

Of course, you can also define the retention (retention) or removal policy (removalPolicy) for the log group.

Ready! The log group of the Lambda function is now encrypted with the KMS key.

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Using Amazon CloudWatch alarms to monitor AWS Lambda

Daniel Bayerlein — Fri, 18 Feb 2022 09:24:33 +0000

You want to receive a notification when your AWS Lambda function does not perform as expected?

In this blog post, I'll show you how to get a notification when your Lambda function hasn't been invoked at least once within a day. You can find the complete AWS CloudFormation template at the end of the post.

Let's start

Besides your Lambda function, you need two other AWS services: Amazon CloudWatch und Amazon SNS.

SNS

Amazon Simple Notification Service is a managed service that provides message delivery from publishers to subscribers.

Since we want an email notification, we set the Protocol property to email and put the recipient email address in the Endpoint property.

Type: AWS::SNS::Topic
Properties:
  Subscription:
    - Endpoint: "add valid email address"
      Protocol: email

📖 Documentation

CloudWatch

Amazon CloudWatch is a monitoring and observability service. You can also add alarms to CloudWatch - let's look at it in detail:

With the AlarmActions property you create the connection to the previously created SNS topic. The Namespace property of the metric associated with the alarm must be set to AWS/Lambda and the Dimensions to the Lambda function. Other properties allow you to specify when the alarm should be triggered.

In this example, the alarm is triggered if the Lambda function has not been invoked once within one day.

Type: AWS::CloudWatch::Alarm
Properties:
  AlarmDescription: Invocation alarm for my AWS Lambda
  AlarmActions:
    - !Ref InvocationAlarmSNSTopic
  Namespace: AWS/Lambda
  MetricName: Invocations
  Dimensions:
    - Name: FunctionName
      Value: !Ref LambdaFunction
  Statistic: Sum
  ComparisonOperator: LessThanThreshold
  Threshold: 1
  EvaluationPeriods: 1
  Period: 86400
  TreatMissingData: breaching

📖 Documentation

CloudFormation template

Here you can find the complete AWS CloudFormation template. Update the CloudWatch Dimensions and the SNS Subscription - and you're ready to go.

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  InvocationAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Invocation alarm for my AWS Lambda
      AlarmActions:
        - !Ref InvocationAlarmSNSTopic
      Namespace: AWS/Lambda
      MetricName: Invocations
      Dimensions:
        - Name: FunctionName
          Value: !Ref LambdaFunction
      Statistic: Sum
      ComparisonOperator: LessThanThreshold
      Threshold: 1
      EvaluationPeriods: 1
      Period: 86400
      TreatMissingData: breaching

  InvocationAlarmSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: "add valid email address"
          Protocol: email

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Array.prototype.includes() can slow down your code

Daniel Bayerlein — Fri, 17 Sep 2021 12:50:55 +0000

In this blog post, I explain when you should avoid Array.prototype.includes() and what you can use instead.

🚀 Not rocket science, or is it?

I ran into a performance issue on a recent project. After some debugging, I came across the following: There was an Array with a large amount of data. To check if a certain value is included Array.prototype.includes() was used. All this is not rocket science - or is it?

⏱ Time for performance measurements

Let's start with a simple measurement. An array with a million entries and we check if certain values are included in the array.

const arr = [...Array(1000000).keys()];

arr.includes(1);        // 0.077ms
arr.includes(10):       // 0.004ms
arr.includes(100);      // 0.003ms
arr.includes(1000);     // 0.003ms
arr.includes(10000);    // 0.014ms
arr.includes(100000);   // 0.113ms
arr.includes(1000000);  // 1.066ms

The value 1000000 is not included in the array - the runtime is already 1 second. Optimizing this is probably still considered micro-optimization. If you use Array.prototype.filter() in combination with Array.prototype.includes() for large data sets, the snail will overtake you!

But why?

The reason for this is the time complexity. Array.prototype.includes() and Array.prototype.filter() has a linear complexity (O(n)).

I found the following article that explains the Big O notation well:

Article No Longer Available

🐇 Almost always as fast as a rabbit

Let's take a look at Set.prototype.has() and compare the performance with Array.prototype.includes().

const arr = [...Array(1000000).keys()];
arr.includes(1000000); // 1.336ms

const arr = [...Array(1000000).keys()];
const setObj = new Set(arr)
setObj.has(1000000); // 0.016ms

Two simple examples with very different runtimes - 1.336ms vs. 0.016ms.

But why?

Set.prototype.has() has a constant complexity (O(1)) meanwhile Array.prototype.includes() has a linear complexity (O(N)).

⏱ More performance measurements

It makes no sense to replace Array.prototype.includes() with Set.prototype.has() everywhere, because it is not always faster. It's important to use functions with loops carefully. 😉

I performed a few benchmarks for this purpose, which you can see in the following table:

Value	`Array.prototype.includes()`	`Set.prototype.has()`	Result
1	836859994.45 ops/s ± 1.01%	176325072.58 ops/s ± 1.49%	`Set.prototype.has()` 78.93% slower
10	826996638.6 ops/s ± 0.95%	87438374.47 ops/s ± 6.73%	`Set.prototype.has()` 89.43% slower
100	800038628.18 ops/s ± 0.56%	143287118.03 ops/s ± 0.86%	`Set.prototype.has()` 82.09% slower
1000	590640746.37 ops/s ± 0.63%	171114526.18 ops/s ± 0.7%	`Set.prototype.has()` 71.03% slower
10000	96545.28 ops/s ± 1.06%	133468419.89 ops/s ± 1.69%	`Array.prototype.includes()` 99.93% slower
100000	9380.42 ops/s ± 0.96%	131819933.56 ops/s ± 0.82%	`Array.prototype.includes()` 99.99% slower

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Deploy a scalable app from scratch in minutes with AWS App Runner

Daniel Bayerlein — Fri, 04 Jun 2021 13:30:55 +0000

Have you heard of AWS App Runner? A few weeks ago, AWS announced a new service called AWS App Runner - the original blog post can be found here. AWS App Runner is a fully managed container application service that let you build, deploy and run containerized web applications and APIs in minutes. I'll show you how to do that in this blog post.

Supported AWS Regions

Currently, AWS App Runner is supported in the following regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo).

Benefits

App Runner resources and infrastructure components are fully managed by AWS, which means you don't have to worry about scaling or infrastructure security or even be an expert in it. Resources are scaled automatically based on traffic. You stay completely focused on your application.

Pricing

You are charged for the compute and memory resources used by your application.
If you automate your deployments you will pay a set monthly fee for each application that covers all automated deployments for that month.
If you opt to deploy from source code, you will pay a build fee for the amount of time it takes App Runner to build a container from your source code.

More information can be found on the AWS App Runner Pricing page.

AWS Console

Of course, you can also deploy the service using AWS CDK. But first, let's take a look at the AWS Console to learn about the service. Because with just a few clicks, your app is ready to go.

Source and deployment

The first step is to choose a source for the App Runner service and how it should be deployed.

Source

You can choose between an AWS Container Registry (Amazon ECR or Amazon ECR Public) and a Git repository provided by GitHub.

If you choose the Container Registry option, then you only need the Container image URI. If you want to use your Git repository on GitHub, you need to install the AWS Connector for GitHub app.

Deployment

Next, you need to choose whether the deployment should be automatic or manual.

Automatic deployment is not available with every source option. If you use Amazon ECR, App Runner monitors your registry and deploys a new version of your service with each image push. If you use your Git repository from GitHub, then each push to the selected branch deploys a new version of your service.

Configure build

This step is only available if you have chosen "Source code repository".

Currently you can choose between Python 3 and Node.js 12 runtime. The "Build command" specify the command that will be used to install the dependencies or compile the code. If you use the configuration file, you can additionally specify pre and post commands! The "Start command" specify the command to start a web server for your service. Also specify the port to be used by the service.

The configuration can be done manually or provided via a configuration file (apprunner.yaml).

version: 1.0
runtime: nodejs12 
build:
  commands:
    pre-build:
      - npm install
    build:
      - npm run build
    post-build:
      - npm test
run: 
  command: npm start
  network:
    port: 3000

📚 App Runner configuration file reference

Configure service

In this step the service is configured by assigning the resources.

Service settings

Name the service and configure the number of CPUs, as well as the required memory.

Auto scaling

You can use the default auto-scaling settings or configure this manually.

If you configure it manually, you can set the following parameters: Concurrency, Minimum size (instances) and Maximum size (instances).

Health check

App Runner performs a TCP health check at the port that your application is listening to.

You can set the parameters Timeout, Interval, Unhealthy threshold and Health threshold.

Security

In the Security section you can select an IAM role for the instance and specify whether to use an AWS-owned key or a custom AWS KMS key.

Deployment time 🚀

In the last step you can review your configuration and then start the creation process! After a short time you will find the URL of your app in the AWS App Runner Console.

In addition to service metrics, event, deployment and application logs are also available in the AWS App Runner Console.

I don't think you can deploy a scalable application any easier or faster. 🚀

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

How to add __typename automatically to your GraphQL document

Daniel Bayerlein — Fri, 19 Feb 2021 07:22:36 +0000

Apollo Client offers the option to automatically adds __typename fields to all outgoing queries.

If you use a minimal GraphQL client like graphql-request, then it is necessary to manually adds __typename fields to all outgoing queries.

After going through the documentation and code of Apollo Client, I became aware of the helper function addTypenameToDocument. This function is responsible to manipulate the GraphQL document by adding __typename automatically. You can use this function with any GraphQL client!

In my case, I wrote a little helper function for graphql-request:

import { addTypenameToDocument } from '@apollo/client/utilities'
import { parse } from 'graphql'
import { GraphQLClient } from 'graphql-request'

const client = new GraphQLClient(endpoint)

export const request = (query, variables, requestHeaders) => {
  // Adds "__typename" to the query
  const document = addTypenameToDocument(
    typeof query === 'string' ? parse(query) : query
  )

  return client.request(document, variables, requestHeaders)
}

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Speed up your Gatsby Build when using GraphCMS

Daniel Bayerlein — Fri, 15 Jan 2021 07:23:35 +0000

Gatsby in combination with GraphCMS is really cool, but the build time can be quite long. I've searched for some improvements, which I would like to share with you in this post.

1. Use Conditional Page Builds

Whenever you run gatsby build, all pages are recreated. You can improve the build time by rebuilding only changed pages.

If you're using GitHub Actions, you can check out my previous post:

Incremental Gatsby Builds with GitHub Actions

Daniel Bayerlein ・ Oct 14 '20

#gatsby #github

Use the environment variable GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES in your gatsby build command to enable conditional page builds and persist the .cache and public directories between builds.

GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby develop

2. Parallel processing of GraphQL queries

The default number of parallel GraphQL queries executed by Gatsby is 4. With the environment variable GATSBY_EXPERIMENTAL_QUERY_CONCURRENCY you can increase the number.

GATSBY_EXPERIMENTAL_QUERY_CONCURRENCY=20 gatsby develop

3. Use Query Batching

By default, gatsby-source-graphql executes each query in a separate network request. The plugin offers a configuration option to batch the requests.

If you set GATSBY_EXPERIMENTAL_QUERY_CONCURRENCY to 20 and maxBatchSize to 5, it means that you are running 4 batches in parallel.

{
  resolve: 'gatsby-source-graphql',
  options: {
    typeName: 'GRAPHCMS',
    fieldName: 'graphCmsData',
    url: 'https://api-eu-central-1.graphcms.com/v2/xxxxxx/master',
    batch: true,
    dataLoaderOptions: {
      maxBatchSize: 5
    }
  }
}

4. Asynchronize your `createPage` function

If you use the createPage function to create dynamic pages with GraphCMS, be sure to run this function asynchronously.

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions
  const result = await graphql(...)

  const pages = result.data.graphCmsData.pages.map(async (page) => {
    createPage({
      path: page.slug,
      component: path.resolve('./src/templates/Page.js'),
      context: {
        id: page.id
      }
    })
  })

  await Promise.all([pages])
}

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

How to handle a multilingual static site with Amazon CloudFront hosted on Amazon S3

Daniel Bayerlein — Thu, 29 Oct 2020 07:31:07 +0000

The easiest way to host a static website on AWS is Amazon S3. In combination with Amazon CloudFront a really fast and cost-effective solution.

But if you run a multilingual static site, you run into the following problem: Amazon CloudFront allows you to specify a default root object, such as index.html, but this will not work in subdirectories.

Take a deeper look into the project structure of the different applications to make them easier to understand.

Single Page Application

my-s3-bucket
├── app.css
├── app.js
└── index.html

Multi Page Application

my-s3-bucket
├── app.css
├── app.js
├── index.html
├── de
│   └── index.html
└── fr
    └── index.html

When you access your CloudFront root URL / in the browser, the requested index.html page is delivered. Everything is fine. But if you call /de instead of /de/index.html in the browser, you will get an error page with the HTTP response status code 403.

A possible solution is to change your links and add /index.html to each link, but I would not recommend this. All right, but which solution is the better one? Use a Lambda function!

Use Lambda@Edge

Lambda@Edge is a feature of Amazon CloudFront that allows your code to run globally at AWS locations close to your users. 🚀

The Lambda function is very simple. Let's have a look at the code:

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const uri = request.uri

  if (uri.endsWith('/')) {
    request.uri += 'index.html'
  } else if (!uri.includes('.')) {
    request.uri += '/index.html'
  }

  callback(null, request)
}

This function inspects the incoming request from the client. Then rewrites the request so that CloudFront requests a default index object (index.html in this case) for each request URI that ends with / or does not contain a dot (.).

❗️ The AWS Lambda function must be deployed in the region us-east-1!

IAM

You also need to create an IAM role that can be assumed by the service principals lambda.amazonaws.com and edgelambda.amazonaws.com. Assign this role to your Lambda function.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "Service": [
               "lambda.amazonaws.com",
               "edgelambda.amazonaws.com"
            ]
         },
         "Action": "sts:AssumeRole"
      }
   ]
}

Update CloudFront settings

Last but not least, it is necessary to set the default root object in the Amazon CloudFront settings to empty, as your Lambda function will handle it.

If you now call /de in the browser, you should see the correct page instead of the error page.

Note: Serverless Framework

As you probably noticed from my previous posts, I use the Serverless Framework in some projects.

If your application is not deployed in the AWS region us-east-1 you will have the following problem: The Serverless Framework does not allow you to deploy individual services in different regions. Therefore it is necessary to use two different serverless.yml manifests.

If you have any kind of feedback, suggestions or ideas - feel free to comment this post!

Forem: Daniel Bayerlein

GraphQL Response Streaming with Amazon API Gateway and GraphQL Yoga

What's new?

GraphQL Server

Using @stream

API Gateway

Ready!

Building robust and scalable serverless event tracking and analytics with AWS

Challenge

Architecture

Infrastructure

Client

Analysis

AWS Glue

Amazon Athena

Run LLMs locally with Ollama on macOS for Developers

Ollama

AI interface

AI Coding Assistant in Visual Studio Code

Malware Protection for S3 with Amazon GuardDuty

Amazon GuardDuty Malware Protection for S3

Code example

Tags

Take action

Tag-based access control (TBAC)

Copy S3 objects

Using pnpm with the GitLab package registry in GitLab CI

Complete code example

Encrypt auto created log groups from AWS Lambda with AWS CDK

AWS KMS

AWS Lambda

Using Amazon CloudWatch alarms to monitor AWS Lambda

Let's start

SNS

CloudWatch

CloudFormation template

Array.prototype.includes() can slow down your code

🚀 Not rocket science, or is it?

⏱ Time for performance measurements

Article No Longer Available

🐇 Almost always as fast as a rabbit

⏱ More performance measurements

Deploy a scalable app from scratch in minutes with AWS App Runner

AWS Console

Source and deployment

Source

Deployment

Configure build

Configure service

Service settings

Auto scaling

Health check

Security

Tags

Deployment time 🚀

How to add __typename automatically to your GraphQL document

Speed up your Gatsby Build when using GraphCMS

1. Use Conditional Page Builds

Incremental Gatsby Builds with GitHub Actions

Daniel Bayerlein ・ Oct 14 '20

2. Parallel processing of GraphQL queries

3. Use Query Batching

4. Asynchronize your createPage function

How to handle a multilingual static site with Amazon CloudFront hosted on Amazon S3

Use Lambda@Edge

Update CloudFront settings

4. Asynchronize your `createPage` function