Forem: Michael Walmsley

Lambda Durable Functions - Keeping your Payloads Secure

Michael Walmsley — Thu, 22 Jan 2026 02:12:48 +0000

AWS Lambda Durable Functions represent a significant evolution in how we build serverless workflows. Instead of managing complex state machines or external orchestrators, developers can write straightforward sequential code that spans minutes, hours, or even days. The framework handles checkpointing, replay, and recovery automatically. However, this convenience introduces a security consideration that deserves careful attention: your workflow data gets persisted in checkpoints, and that data may contain sensitive information.

In this article we will explore the deeper parts of the Lambda Durable SDK and explore how you can go about securing your precious payload data.

What Are Durable Functions?

Lambda Durable Functions enable you to write long-running workflows as regular Lambda code. The durable execution SDK provides primitives like steps, waits, and callbacks that automatically create checkpoints of your function's progress. When your function pauses for a wait operation or gets interrupted by a failure, Lambda can resume exactly where it left off by replaying the checkpointed operations.

The checkpoint and replay mechanism works by storing the results of each durable operation. When your function executes a step that calls an external API, the SDK captures that step's return value in a checkpoint before continuing. If your function later pauses or fails, the next invocation replays from the beginning but uses the stored checkpoint results instead of re-executing completed steps. This approach allows your code to remain deterministic while spanning execution times far beyond Lambda's 15-minute invocation limit.

For a comprehensive introduction to durable functions, I recommend reading the AWS Lambda durable functions announcement and reviewing the official documentation.

Eric Johnson has a great series of articles covering Lambda Durable Functions:

Developing AWS Lambda Durable Functions with AWS SAM

AWS Lambda Durable Functions: Build Workflows That Last

Is this code deterministic?

Where Checkpoint Data Lives and the Security Model

Understanding where your checkpoint data gets stored is essential for implementing proper security controls. Lambda stores durable execution state in an AWS-managed service infrastructure. When you enable durable execution on a Lambda function, AWS manages all the checkpoint storage behind the scenes. Your function's execution role requires permissions for lambda:CheckpointDurableExecution and lambda:GetDurableExecutionState to interact with this storage, and a function execution is only able to access its own checkpoint data. The actual data lives in AWS-managed infrastructure, not in resources within your AWS account.

Each checkpoint contains the operation type, operation name, input parameters, output values, and metadata like timestamps and retry counts. For a payment processing workflow, this means the checkpoint storage contains customer payment details, credit card information, or other personally identifiable information depending on what data flows through your durable operations.

Lambda automatically encrypts checkpoint data at rest using AWS-owned encryption keys and in transit using TLS when Lambda reads or writes checkpoints. However, AWS does not currently support using customer-managed KMS keys (CMK) for checkpoint storage. You cannot control key rotation, access policies, or audit logging for the storage encryption itself. For many applications, this level of security is perfectly adequate. The data remains encrypted at rest and in transit, and access is controlled through IAM permissions on the checkpoint APIs.

However, organizations with strict compliance requirements around encryption key management need additional controls beyond what the default checkpoint storage provides. Healthcare applications subject to HIPAA, financial services handling payment card data under PCI-DSS, or government systems with data sovereignty requirements often mandate customer-controlled encryption keys. This is where custom serialization becomes essential and the Lambda Durable Functions team have provided SDK hooks for custom serialisation and deserialisation (SerDes) for payload checkpointing.

The AWS documentation explicitly states that when you implement custom encryption through serializers and deserializers, you lose visibility of operation results in the Lambda console and API responses. Checkpoint data appears encrypted in execution history and cannot be inspected without decryption. This trade-off between security and operational visibility is unavoidable with the current architecture. You must decide which is more important for your use case.

Full Payload Encryption

The recommendation is implementing custom encryption using the SDK's SerDes mechanism when you need additional security controls beyond the default encryption (source). This approach encrypts the entire step result before it reaches the checkpoint storage. You maintain full control over the encryption keys, can use customer-managed KMS keys, and can implement any encryption algorithm your compliance requirements demand.

The implementation involves creating a custom serializer class that encrypts data during checkpoint creation and decrypts it during replay.

TypeScript SerDes Implementation

import type { Serdes } from '@aws/durable-execution-sdk-js';
import { DecryptCommand, EncryptCommand, KMSClient } from '@aws-sdk/client-kms';

const kmsClient = new KMSClient({});
const KMS_KEY_ID = process.env.KMS_KEY_ID;

// SerdesContext interface from the SDK (not exported in v1.0.1)
interface SerdesContext {
  entityId: string;
  durableExecutionArn: string;
}

export class KmsSerDer<T> implements Serdes<T> {
  async serialize(
    value: T | undefined,
    context: SerdesContext,
  ): Promise<string | undefined> {
    if (value === undefined) {
      return undefined;
    }

    if (!KMS_KEY_ID) {
      throw new Error('KMS_KEY_ID environment variable is not set');
    }

    const plaintext = JSON.stringify(value);
    const encoder = new TextEncoder();
    const plaintextBytes = encoder.encode(plaintext);

    const command = new EncryptCommand({
      KeyId: KMS_KEY_ID,
      Plaintext: plaintextBytes,
      EncryptionContext: {
        entityId: context.entityId,
        durableExecutionArn: context.durableExecutionArn,
      },
    });

    const response = await kmsClient.send(command);

    if (!response.CiphertextBlob) {
      throw new Error('Encryption failed: no ciphertext returned');
    }

    // Convert to base64 for storage
    return Buffer.from(response.CiphertextBlob).toString('base64');
  }

  async deserialize(
    encryptedData: string | undefined,
    context: SerdesContext,
  ): Promise<T | undefined> {
    if (encryptedData === undefined) {
      return undefined;
    }

    // Convert from base64
    const ciphertextBlob = Buffer.from(encryptedData, 'base64');

    const command = new DecryptCommand({
      CiphertextBlob: ciphertextBlob,
      EncryptionContext: {
        entityId: context.entityId,
        durableExecutionArn: context.durableExecutionArn,
      },
    });

    const response = await kmsClient.send(command);

    if (!response.Plaintext) {
      throw new Error('Decryption failed: no plaintext returned');
    }

    const decoder = new TextDecoder();
    const plaintext = decoder.decode(response.Plaintext);

    return JSON.parse(plaintext) as T;
  }
}

Applying SerDes to a Durable Step

Once you have created a class implementing the SerDes interface (shown above) you can apply that to each step where you want custom serialisation to be used.

    // Use KMS encryption for the order step
    const order = await context.step<Order>(
      'create-order',
      async () => {
        return createOrder(event);
      },
      {
        serdes: kmsSerDes,
      },
    );

The example implementation makes direct use of KMS keys which have an upper limit of 4KB on payload size for encryption, which may not be suitable for all payload types. Where you need to encrypt more than 4KB of data you will need to make use of the AWS encryption SDK and use envelope encryption.

Envelope encryption is where your data is encrypted with a unique data encryption key (DEK), and that DEK is then encrypted with your KMS master key. This approach means you can encrypt large amounts of data quickly with symmetric encryption while KMS only needs to encrypt/decrypt the small DEK, and the encrypted message contains both the encrypted data and the encrypted DEK together in one portable blob.

In the github link at the end of this article I have provided examples of both implementations - direct KMS and AWS Envelope encryption.

What This Looks Like in Production

When you view the durable execution in the Lambda console, the checkpoint data appears as an encrypted blob. The execution timeline shows that steps completed successfully, but you cannot see the actual data without decrypting it which reduces operational visibility and increases support friction.

Wthout Encryption

{
  "EventId": 3,
  "EventTimestamp": "2026-01-21T12:53:09.678Z",
  "EventType": "StepSucceeded",
  "Id": "c4ca4238a0b92382",
  "Name": "create-order",
  "StepSucceededDetails": {
    "RetryDetails": {
      "CurrentAttempt": 1
    },
    "Result": {
      "Truncated": false,
      "Payload": "{\"customer\":{\"customer_id\":\"123456789\",\"name\":\"John Doe\",\"email\":\"john@example.com\",\"ssn\":\"123-45-6789\",\"address\":\"123 Main St\"},\"payment\":{\"method\":\"credit_card\",\"creditCard\":\"4111-1111-1111-1111\",\"amount\":100},\"items\":[{\"id\":\"123\",\"quantity\":2},{\"id\":\"456\",\"quantity\":2}]}"
    }
  },
  "SubType": "Step"
}

With Encryption

{
  "EventId": 3,
  "EventTimestamp": "2026-01-21T13:46:04.182Z",
  "EventType": "StepSucceeded",
  "Id": "c4ca4238a0b92382",
  "Name": "create-order",
  "StepSucceededDetails": {
    "RetryDetails": {
      "CurrentAttempt": 1
    },
    "Result": {
      "Truncated": false,
      "Payload": "AQICAHjFMg8rj2ThqXWiaabEbpqoKCF9xVaTOZP6HOCD17VYhwHddxCX5VW6qI5ehwq4zFGyAAABADCB/QYJKoZIhvcNAQcGoIHvMIHsAgEAMIHmBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOc8WQxUXh32Isp/sQIBEICBuJWXZ3ST7HPxVuZH3epnhKwcyr2llFzKlUQrzxHxbGkqvNc9o2uhsbzGCq0Lnxq5uAa8aVRYR3Q7ijoGWctSa0NdAIv1FxiUlk9lh7fOALzZuifPxrmcQL4LZTl/ZQBje9GUfNZ9bhI4UbdayCqD8tY6CdYzEWzI2xIoBlAPHdwSX4Y9MGVGFJD8kyhb/oSdJBQmqieqT1DcGsxAAOYEKEtsL/V9YOC4NbxKgnyR7j4YGg+Y+2ImvdU="
    }
  },
  "SubType": "Step"
}

You can see that your workflow progressed through its steps, but the sensitive data remains encrypted and un-readable. This provides strong security guarantees at the cost of operational visibility. When troubleshooting a failed workflow, you cannot simply look at the console to see what data was being processed. You must decrypt the checkpoint data programmatically, which requires appropriate KMS permissions.

Field-Level Encryption: Preserving Structure for Operations

Full payload encryption solves the security problem but introduces a significant operational challenge. Once encrypted, your checkpoint data becomes completely opaque. You cannot determine which customer's order failed. You cannot identify patterns in failures across different payment amounts. The lack of visibility makes production operations substantially harder.

A more sophisticated approach involves encrypting only the sensitive fields while preserving the overall structure of your data. This technique provides the security benefits of encryption while maintaining the operational visibility needed for production systems. The key insight is that not all data in your checkpoints requires encryption. Order IDs, timestamps, workflow states, and similar metadata can remain in plaintext for operational purposes.

Field-level encryption works by replacing sensitive fields with placeholder objects and encrypting all the sensitive values together in a single operation. The encrypted blob and the modified structure both get stored in the checkpoint. When the workflow resumes, the SDK automatically decrypts the blob and restores the original field values. Despite encrypting multiple fields, this approach makes only a single KMS call, maintaining the performance benefit of batch encryption.

TypeScript Implementation

import type { Serdes } from '@aws/durable-execution-sdk-js';
import {
  buildClient,
  CommitmentPolicy,
  KmsKeyringNode,
} from '@aws-crypto/client-node';

const KMS_KEY_ARN = process.env.KMS_KEY_ARN;

// SerdesContext interface from the SDK (not exported in v1.0.1)
interface SerdesContext {
  entityId: string;
  durableExecutionArn: string;
}

interface EncryptedPayload<T> {
  data: T;
  __encrypted_pii?: string;
}

/**
 * Field-level encryption SerDes using AWS Encryption SDK with envelope encryption
 * This provides better performance for large payloads by using data keys
 */
export class EnvelopeEncryptionSerDes<T> implements Serdes<T> {
  private fieldPaths: string[];
  private encryptionContext: Record<string, string>;
  private client: ReturnType<typeof buildClient>;
  private keyring: KmsKeyringNode | null = null;

  /**
   * @param fieldPaths - Array of dot-notation paths to fields that should be encrypted
   *                     e.g., ['customer.ssn', 'payment.creditCard']
   * @param encryptionContext - Additional encryption context
   */
  constructor(
    fieldPaths: string[],
    encryptionContext: Record<string, string> = {},
  ) {
    this.fieldPaths = fieldPaths;
    this.encryptionContext = encryptionContext;

    // Build the encryption client with commitment policy
    this.client = buildClient(CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT);
  }

  private getKeyring(): KmsKeyringNode {
    if (!this.keyring) {
      if (!KMS_KEY_ARN) {
        throw new Error('KMS_KEY_ARN environment variable is not set');
      }

      // Create KMS keyring for envelope encryption
      this.keyring = new KmsKeyringNode({
        generatorKeyId: KMS_KEY_ARN,
      });
    }
    return this.keyring;
  }

  async serialize(
    value: T | undefined,
    context: SerdesContext,
  ): Promise<string | undefined> {
    if (value === undefined) {
      return undefined;
    }

    // Clone the object to avoid mutating the original
    const clonedValue = JSON.parse(JSON.stringify(value));

    // Extract sensitive fields
    const sensitiveData: Record<string, unknown> = {};
    for (const path of this.fieldPaths) {
      const fieldValue = this.getNestedValue(clonedValue, path);
      if (fieldValue !== undefined) {
        sensitiveData[path] = fieldValue;
        // Replace with marker
        this.setNestedValue(clonedValue, path, { __encrypted: path });
      }
    }

    // Encrypt the sensitive data if any fields were found
    let encryptedPii: string | undefined;
    if (Object.keys(sensitiveData).length > 0) {
      const plaintext = JSON.stringify(sensitiveData);

      // Build encryption context
      const fullContext = {
        entityId: context.entityId,
        durableExecutionArn: context.durableExecutionArn,
        ...this.encryptionContext,
      };

      // Encrypt using AWS Encryption SDK (envelope encryption)
      const { result } = await this.client.encrypt(
        this.getKeyring(),
        plaintext,
        {
          encryptionContext: fullContext,
        },
      );

      // Convert to base64 for storage
      encryptedPii = Buffer.from(result).toString('base64');
    }

    // Build the final payload
    const payload: EncryptedPayload<unknown> = {
      ...clonedValue,
    };

    if (encryptedPii) {
      payload.__encrypted_pii = encryptedPii;
    }

    return JSON.stringify(payload);
  }

  async deserialize(
    encryptedData: string | undefined,
    context: SerdesContext,
  ): Promise<T | undefined> {
    if (encryptedData === undefined) {
      return undefined;
    }

    const payload: EncryptedPayload<unknown> = JSON.parse(encryptedData);

    // If there's encrypted PII, decrypt it
    if (payload.__encrypted_pii) {
      const ciphertext = Buffer.from(payload.__encrypted_pii, 'base64');

      // Decrypt using AWS Encryption SDK
      const { plaintext, messageHeader } = await this.client.decrypt(
        this.getKeyring(),
        ciphertext,
      );

      // Verify encryption context matches
      const expectedContext = {
        entityId: context.entityId,
        durableExecutionArn: context.durableExecutionArn,
        ...this.encryptionContext,
      };

      for (const [key, value] of Object.entries(expectedContext)) {
        if (messageHeader.encryptionContext[key] !== value) {
          throw new Error(
            `Encryption context mismatch for key ${key}: expected ${value}, got ${messageHeader.encryptionContext[key]}`,
          );
        }
      }

      const plaintextString = plaintext.toString('utf8');
      const sensitiveData: Record<string, unknown> =
        JSON.parse(plaintextString);

      // Restore the sensitive fields
      for (const [path, value] of Object.entries(sensitiveData)) {
        this.setNestedValue(payload, path, value);
      }

      // Remove the encryption metadata
      delete payload.__encrypted_pii;
    }

    return payload as T;
  }

  private getNestedValue(obj: unknown, path: string): unknown {
    const parts = path.split('.');
    let current = obj;
    for (const part of parts) {
      if (current === undefined || current === null) {
        return undefined;
      }
      current = (current as Record<string, unknown>)[part];
    }
    return current;
  }

  private setNestedValue(obj: unknown, path: string, value: unknown): void {
    const parts = path.split('.');
    let current = obj as Record<string, unknown>;
    for (let i = 0; i < parts.length - 1; i++) {
      const part = parts[i];
      if (current[part] === undefined || current[part] === null) {
        current[part] = {};
      }
      current = current[part] as Record<string, unknown>;
    }
    current[parts[parts.length - 1]] = value;
  }
}

Applying this to Workflow Steps

  // Create dedicated SerDes using AWS Encryption SDK with envelope encryption
  const orderSerDes = new EnvelopeEncryptionSerDes<Order>(
    ['customer.ssn', 'payment.creditCard'],
    {
      service: 'order-processing',
      environment: 'production',
    },
  );

  ...

  // Use envelope encryption (AWS Encryption SDK) for the order step
  // Only customer.ssn and payment.creditCard will be encrypted
  const order = await context.step<Order>(
    'create-order',
    async () => {
      return createOrder(event);
    },
    {
      serdes: orderSerDes,
    },
  );

What Field-Level Encryption Looks Like in the Console

With field-level encryption, the Lambda console displays a much more useful view of your checkpoint data. The execution history shows the workflow structure with sensitive fields clearly marked as encrypted.

{
  "EventId": 3,
  "EventTimestamp": "2026-01-21T14:29:08.022Z",
  "EventType": "StepSucceeded",
  "Id": "c4ca4238a0b92382",
  "Name": "create-order",
  "StepSucceededDetails": {
    "RetryDetails": {
      "CurrentAttempt": 1
    },
    "Result": {
      "Truncated": false,
      "Payload": "{\"id\":\"order-1769005747947\",\"customerId\":\"123456789\",\"customer\":{\"customer_id\":\"123456789\",\"name\":\"John Doe\",\"email\":\"john@example.com\",\"ssn\":{\"__encrypted\":\"customer.ssn\"},\"address\":\"123 Main St\"},\"payment\":{\"method\":\"credit_card\",\"creditCard\":{\"__encrypted\":\"payment.creditCard\"},\"amount\":100},\"items\":[{\"id\":\"123\",\"quantity\":2},{\"id\":\"456\",\"quantity\":2}],\"createdAt\":\"2026-01-21T14:29:07.947Z\",\"__encrypted_pii\":\"AgV4x2uTyXhneXm5Lt2L3qRfCcsJ0WCJZxgt8uLZdSLxp9EBjgAFABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREE1aFpLbWdvUk5RYU9PM1h2VThnRjFqRW9raXErZXpBZUlVM2dnVmZxTDBpTHlOYWpNZFlzRXM3dE05YVdkUTl5dz09ABNkdXJhYmxlRXhlY3V0aW9uQXJuANdhcm46YXdzOmxhbWJkYTphcC1zb3V0aGVhc3QtMjo4NTk1Nzg2NTExMDI6ZnVuY3Rpb246TGFtYmRhRHVyYWJsZUZ1bmN0aW9uRXhhbS1FbnZlbG9wZVdvcmtmbG93RnVuY3Rpb24teTNPNndQM1d4bXBkOiRMQVRFU1QvZHVyYWJsZS1leGVjdXRpb24vYzkzNTk5ODgtMDQ4ZS00NTE5LWE0YTctNmU2ZWI2YzYwNjU4LzkwNWVkNjQ3LTc5OTctM2JmNi05Y2M4LWZiYjVkMTBhYjQ0YwAIZW50aXR5SWQAATEAC2Vudmlyb25tZW50AApwcm9kdWN0aW9uAAdzZXJ2aWNlABBvcmRlci1wcm9jZXNzaW5nAAEAB2F3cy1rbXMAUGFybjphd3M6a21zOmFwLXNvdXRoZWFzdC0yOjg1OTU3ODY1MTEwMjprZXkvYzVmNGNkYzMtMWYyMy00NDUyLWEwMDAtZTdjNDgxODM3MTliALgBAgEAeMUyDyuPZOGpdaJppsRumqgoIX3FVpM5k/oc4IPXtViHAQ4ueB4Z5NN/r6oUnaYcKKQAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwo28yWtu49aAyYS2ACARCAO825YL1fsuSZUxS1VcxP1CgsTtoIfc+rtwUVUKgSh1+UwvXxZTxN7RVEPu/IK7GIGY9QsDi3zPV6zxUaAgAAEAAebuVheeSBpQIdicZSiFGZliUt3IPJxLhFQ4974/vNuEWWFb7JD1RZNub7/iCA++7/////AAAAAQAAAAAAAAAAAAAAAQAAAEk4GWvzu2BZDrdKhUKFM3zR1A6b2I/0DR/Bfs+gdGWfBsn+m4CCCq/zsYElqc8wGroQS3bd1m3+0+Qea+wwRGv2vPBJD7RldQQPfNr4AwI22ckfiFQYwH65eABoMGYCMQCI+jB0SXbxSQs9Pyvil1sjjM0kkZ+YhWyxcirggWnLzE43xNfB2U2IarmN+k0kE+wCMQDSbQSTleKz4yzJG/AXcyUQ2ohcFFtYZcOzlsImr1a7tRgOfrJ2sRxYhQF64tPtMrw=\""
    }
  },
  "SubType": "Step"
}

JSON formatted Payload

Encrypted fields have been replaced with an enryption placeholder rather than showing the actual data. The encrypted data is still in the payload but at the end in a single blob, the overall payload structure is maintained.

Using placeholder text enables robust decryption to occur using the actual object names rather than relying on order of fields which can be haphazard.

{
  "id": "order-1769005747947",
  "customerId": "123456789",
  "customer": {
    "customer_id": "123456789",
    "name": "John Doe",
    "email": "john@example.com",
    "ssn": {
      "__encrypted": "customer.ssn"
    },
    "address": "123 Main St"
  },
  "payment": {
    "method": "credit_card",
    "creditCard": {
      "__encrypted": "payment.creditCard"
    },
    "amount": 100
  },
  "items": [
    {
      "id": "123",
      "quantity": 2
    },
    {
      "id": "456",
      "quantity": 2
    }
  ],
  "createdAt": "2026-01-21T14:29:07.947Z",
  "__encrypted_pii": "AgV4x2uTyXhneXm5Lt2L3qRfCcsJ0WCJZxgt8uLZdSLxp9EBjgAFABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREE1aFpLbWdvUk5RYU9PM1h2VThnRjFqRW9raXErZXpBZUlVM2dnVmZxTDBpTHlOYWpNZFlzRXM3dE05YVdkUTl5dz09ABNkdXJhYmxlRXhlY3V0aW9uQXJuANdhcm46YXdzOmxhbWJkYTphcC1zb3V0aGVhc3QtMjo4NTk1Nzg2NTExMDI6ZnVuY3Rpb246TGFtYmRhRHVyYWJsZUZ1bmN0aW9uRXhhbS1FbnZlbG9wZVdvcmtmbG93RnVuY3Rpb24teTNPNndQM1d4bXBkOiRMQVRFU1QvZHVyYWJsZS1leGVjdXRpb24vYzkzNTk5ODgtMDQ4ZS00NTE5LWE0YTctNmU2ZWI2YzYwNjU4LzkwNWVkNjQ3LTc5OTctM2JmNi05Y2M4LWZiYjVkMTBhYjQ0YwAIZW50aXR5SWQAATEAC2Vudmlyb25tZW50AApwcm9kdWN0aW9uAAdzZXJ2aWNlABBvcmRlci1wcm9jZXNzaW5nAAEAB2F3cy1rbXMAUGFybjphd3M6a21zOmFwLXNvdXRoZWFzdC0yOjg1OTU3ODY1MTEwMjprZXkvYzVmNGNkYzMtMWYyMy00NDUyLWEwMDAtZTdjNDgxODM3MTliALgBAgEAeMUyDyuPZOGpdaJppsRumqgoIX3FVpM5k/oc4IPXtViHAQ4ueB4Z5NN/r6oUnaYcKKQAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwo28yWtu49aAyYS2ACARCAO825YL1fsuSZUxS1VcxP1CgsTtoIfc+rtwUVUKgSh1+UwvXxZTxN7RVEPu/IK7GIGY9QsDi3zPV6zxUaAgAAEAAebuVheeSBpQIdicZSiFGZliUt3IPJxLhFQ4974/vNuEWWFb7JD1RZNub7/iCA++7/////AAAAAQAAAAAAAAAAAAAAAQAAAEk4GWvzu2BZDrdKhUKFM3zR1A6b2I/0DR/Bfs+gdGWfBsn+m4CCCq/zsYElqc8wGroQS3bd1m3+0+Qea+wwRGv2vPBJD7RldQQPfNr4AwI22ckfiFQYwH65eABoMGYCMQCI+jB0SXbxSQs9Pyvil1sjjM0kkZ+YhWyxcirggWnLzE43xNfB2U2IarmN+k0kE+wCMQDSbQSTleKz4yzJG/AXcyUQ2ohcFFtYZcOzlsImr1a7tRgOfrJ2sRxYhQF64tPtMrw="
}

The __encrypted_pii field contains the secure JSON data as an object.

{
  "customer.ssn": "123-45-6789",
  "payment.creditCard": "4111-1111-1111-1111"
}

The JSON is then:

Stringified to plaintext
Encrypted with KMS (produces binary ciphertext)
Base64 encoded for JSON storage

The operational benefits become immediately clear. You can see the order ID, customer name, email, and payment amount in plaintext. When troubleshooting a failed payment, you can identify the specific order and see the payment amount without needing to decrypt anything. You can identify patterns like failures occurring only for payments above certain amounts. The sensitive fields are clearly marked with the __encrypted placeholder, making it obvious what data is protected while keeping everything else visible for operational purposes.

Making This Readily Available

The field-level encryption pattern solves a real problem that many developers face when building production durable functions. Rather than requiring every team to implement their own version, this functionality belongs in a reusable utility. I have submitted an RFC proposal to extend the DataMasking utility in Python to include field level encryption and an RFC for Typescript Powertools to implement and expose the same utility with full feature parity (Data Masking not available in Typescript Powertools).

If you find this functionality valuable for your durable functions, I encourage you to review and support the RFCs. Your feedback and use cases will help shape the final implementation. Visit the Python RFC and TypeScript RFC to add your voice to the discussion. A thumbs up reaction on the RFC helps demonstrate community demand and prioritization for the Powertools maintainers.

The security of your checkpoint data matters. Whether you choose full encryption for maximum security or field-level encryption for operational visibility, having a well-tested, reusable implementation will make securing your durable functions significantly easier. The examples in this article provide working code you can use today, and the Powertools RFCs aim to make this functionality available as a maintained, tested utility for the entire serverless community.

Full source code is available on GitHub https://github.com/walmsles/lambda-durable-function-encryption

Run Any MCP Server Securely Without Changing Its Config

Michael Walmsley — Tue, 06 Jan 2026 12:55:43 +0000

MCP servers are powerful - they give Claude Desktop or other AI assistants access to your filesystem, databases, APIs, and more. But that power comes with risk. Every MCP server runs with your full user permissions. A bug or malicious MCP package could read your SSH keys, AWS credentials, or browser cookies.

When you add an MCP server to Claude Desktop, you're running arbitrary code on your machine. That filesystem server? It can see everything your user can see. That database tool? It has access to your entire home directory. Most of us just... trust it. We copy the config from the README and hope for the best, because we want immediate results.

The Traditional Container Approach

Some security-conscious users already run MCP servers in containers using Docker, or other container runtimes. Here's what that looks like:

{
  "mcpServers": {
    "sqlite": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-v", "/home/michael/data:/data",
        "-v", "/home/michael/.cache/uv:/home/mcp/.cache/uv",
        "-e", "HOME=/home/mcp",
        "--user", "1000:1000",
        "ghcr.io/serverlessdna/mcp-python:latest",
        "uvx", "mcp-server-sqlite", "--db-path", "/data/mydb.sqlite"
      ]
    }
  }
}

That's 12 lines of Docker arguments you need to get right and it looks nothing like the example configuration the MCP server provider gives you. Volume mounts, user permissions, environment variables, image tags. Miss one flag and it fails silently or breaks in subtle ways.

Most people look at that and think "I'll just run it natively".

What if you could run MCP servers in complete isolation, with access only to what you explicitly allow - and configure them exactly as their READMEs document?

Most MCP server documentation shows something like this:

{
  "mcpServers": {
    "sqlite": {
      "command": "uvx",
      "args": ["mcp-server-sqlite", "--db-path", "~/data/mydb.sqlite"]
    }
  }
}

With run-mcp, your config is almost the same:

{
  "mcpServers": {
    "sqlite": {
      "command": "run-mcp",
      "args": ["uvx", "mcp-server-sqlite", "--db-path", "/data/mydb.sqlite"],
      "env": {
        "MCP_MOUNT": "~/data:/data"
      }
    }
  }
}

One word changes. Replace uvx or npx with run-mcp, add the original command as the first argument, and declare what the server can access. That's it.

Three approaches, compared:

Approach	Lines of Config	Container Isolation	Complexity
Native (uvx/npx)	4	❌ None	Low
Raw Docker	12+	✅ Full	High
run-mcp	5	✅ Full	Low

The server runs in a container with zero access to your host - unless you explicitly grant it through a registered mount using the MCP_MOUNT environment variable as shown in the example.

What Changes

Without run-mcp	With run-mcp
Server sees entire home directory	Server sees only what you mount
Server can read `~/.aws`, `~/.ssh`	No access unless you add it
Malicious package = full compromise	Malicious package = contained
Servers share your environment	Each server gets isolated storage

The security model flips from "access everything by default" to "access nothing by default."

How It Works

Replace the command - uvx becomes run-mcp uvx, npx becomes run-mcp npx
Mount what you need - MCP_MOUNT=~/data:/data grants explicit access.
Credentials are opt-in - Add ~/.aws:/home/mcp/.aws:ro only when the server needs AWS access and provides read-only access.

No Docker knowledge required. No Dockerfile. No docker-compose. Just a single binary that handles everything across all platforms (Windows, macOS, Linux).

Real Examples

Filesystem Server (Read-Only Access)

{
  "mcpServers": {
    "filesystem": {
      "command": "run-mcp",
      "args": ["npx", "@modelcontextprotocol/server-filesystem", "/docs"],
      "env": {
        "MCP_MOUNT": "~/Documents:/docs:ro"
      }
    }
  }
}

The server can only read your Documents folder. Nothing else.

AWS Server (With Credentials)

{
  "mcpServers": {
    "aws-api": {
      "command": "run-mcp",
      "args": ["uvx", "awslabs.aws-api-mcp-server"],
      "env": {
        "MCP_MOUNT": "~/.aws:/home/mcp/.aws:ro",
        "AWS_REGION": "us-east-1"
      }
    }
  }
}

The server gets read-only access to your AWS credentials. Nothing else on your system is visible.

Memory Server (No Host Access)

{
  "mcpServers": {
    "memory": {
      "command": "run-mcp",
      "args": ["npx", "@modelcontextprotocol/server-memory"]
    }
  }
}

No MCP_MOUNT means no host filesystem access at all. The server gets an isolated home directory that persists between runs - but it's completely sandboxed using standard volume features of container runtimes.

No Runtime Dependencies

Don't have Node.js 22 installed? Python 3.12? Doesn't matter.

run-mcp auto-detects your container runtime - Docker, Podman - and runs the appropriate container. Your machine stays clean. No version conflicts. No global package pollution.

Getting Started

Install the binary:

# Linux (amd64)
curl -fsSL https://github.com/serverless-dna/run-mcp/releases/latest/download/run-mcp-linux-amd64 \
  -o ~/.local/bin/run-mcp && chmod +x ~/.local/bin/run-mcp

# macOS (Apple Silicon)
curl -fsSL https://github.com/serverless-dna/run-mcp/releases/latest/download/run-mcp-darwin-arm64 \
  -o ~/.local/bin/run-mcp && chmod +x ~/.local/bin/run-mcp

# macOS (Intel)
curl -fsSL https://github.com/serverless-dna/run-mcp/releases/latest/download/run-mcp-darwin-amd64 \
  -o ~/.local/bin/run-mcp && chmod +x ~/.local/bin/run-mcp

# Windows (PowerShell)
New-Item -ItemType Directory -Force -Path ~\.local\bin | Out-Null
Invoke-WebRequest -Uri https://github.com/serverless-dna/run-mcp/releases/latest/download/run-mcp-windows-amd64.exe -OutFile ~\.local\bin\run-mcp.exe

Note: Ensure ~/.local/bin is in your PATH, or use the full path to run-mcp in your Claude Desktop config.

Update your Claude Desktop config and you are done.

The Security Boundary That Should Have Been There

Every MCP server you run is code executing on your machine with your permissions. The MCP protocol is powerful, but it shipped without a security model for the host system.

run-mcp adds that missing layer. Same simple configs. Same MCP servers. Now with isolation by default.

Links:

Questions or feedback? Open an issue or find me on LinkedIn

Web Sockets Don't Pub-Sub

Michael Walmsley — Sun, 02 Mar 2025 12:26:22 +0000

Web Sockets have become a fundamental technology in modern web development, but their true function is often misunderstood. The killer use-case for Web Sockets has been in the development and proliferation of chat apps where users jump into a channel and write messages to one another. This has led to a wide spread understanding that Web Sockets provide publish and subscribe capability for your application - but this is a myth - Web Sockets do not Pub-Sub. In this article you will learn about Web Sockets, what they do and how they fit into your application architecture and common uses for Web Sockets in web application development.

What are Web Sockets?

Web Sockets represent a communication protocol that creates a persistent connection between a client (typically a web browser) and a server. Unlike traditional HTTP, which follows a request-response pattern, Web Sockets allow:

Two-way communication: Both server and client can send messages at any time
Persistent connection: One connection stays open instead of creating new ones for each interaction.
Low latency: Messages transmit immediately without the overhead of establishing new connections

Before Web Sockets, developers used techniques like HTTP polling, where browsers would repeatedly ask servers "Do you have anything new for me?". This is inefficient and causes delays. Web Sockets solved this problem by creating a direct communication channel that stays open enabling servers to send data directly to clients as soon as it is available which enables powerful real-time bi-directional communication.

How Web Sockets Work?

WebSocket connections are an instance of an upgraded standard HTTP connection over TCP. When connecting to a web server with the “wss” protocol - the server attempts to upgrade the connection from a standard HTTP web connection to a persistent websocket one.

This process happens in two main phases:

The Handshake

When a client wants to establish a WebSocket connection:

Initial Request: The client sends an HTTP request that says "I'd like to upgrade this connection to Web Sockets" and ”I support these sub-protocols”.
Server Verification: The server responds with a special code confirming it accepts the WebSocket connection and also confirms the sub-protocol it prefers the client to use.
Connection Established: Once the handshake completes successfully, the HTTP connection "upgrades" to a WebSocket connection.

This handshake is designed to be compatible with HTTP infrastructure, so Web Sockets can work on standard web ports (80 and 443).

Data Transfer

After the connection is established:

Data is sent in units called "frames".
Messages can be text (UTF-8) or binary data.
Either side can send messages at any time and messages must conform to the agreed sub-protocol.
For security, all client-to-server messages are encrypted over the TSL channel created when connecting.
The connection stays open until either side decides to close it.

It is important to understand that with a Web Socket connection both the client and server are able to send messages to the other at anytime. Any data written to the Web Socket will be transported to the other party. This is where sub-protocols become important and are a critical component of the design process for an application leveraging this technology. The way clients and servers communicate over Web Sockets is defined by the sub-protocol.

What are SubProtocols?

Subprotocols are agreed-upon conventions for how messages should be formatted and interpreted. Think of them as languages that both client and server agree to speak. Sub-protocols are important for:

Standardization: Provide consistent ways to format messages (like JSON, XML, or custom formats)
Messaging Syntax: provides the order and meaning of messages and defines any request/response style interactions and how these are tracked. If you are using web-sockets to transfer request/response interactions these need to be carefully thought out and designed so the client and server can correlate related messages.
Versioning: Allows protocols to evolve while maintaining backward compatibility
Interoperability: Different implementations can work together if they follow the same subprotocol

During the handshake, the client can say "I can speak these sub-protocols" and the server responds with "Let's use this one." For example:

A chat application might use a chat-specific sub-protocol.
A financial application might use a different sub-protocol optimized for market data.
A game might use yet another sub-protocol designed for real-time player movements.

By eliminating the constant request-response cycle and providing a direct communication channel, Web Sockets make interactive web applications more responsive and efficient, especially when real-time updates are essential.

Where is the Pub-Sub?

The Web Sockets layer itself doesn’t provide any capability or functionality other than handling the client connections and transferring data between the client and server applications that sit at either end of the data pipe connection (Web Socket).

The connections starts as a normal HTTP connection with DNS Lookup and TLS handshake for a secure connection. Once the HTTP connection is complete the client will send a standard HTTP get asking to upgrade the connection to a websocket. It is at this point the client will include a list of sub-protocols it understands. The server provides the client sub-protocols and asks the backend server to authorize the connection. The backend server will respond with the sub-protocol is prefers to use from those provided by the client and whether the connection is authorized to continue.

It is at this point that the connection is now established and either the client or server are able to send data to each other using the selected sub-protocol. The client is able to send a message conforming to the sub-protocol syntax to the server for processing. Periodically the Web Socket server may send keep-alive messages which are documented as part of the Web Socket protocol - these "Ping"-"Pong" messages are a way to ensure the connection is healthy.

Web Sockets, being bi-directional, enables the server to also send notifications to the client. These can occur at anytime and the client decides how to handle these. The sub-protocol determines the structure of data to be sent between the client and server and it also dictates how each actor will respond to messages being sent.

When the client and server communications are complete, either may request to close down the connection and disconnect.

Web Sockets do not provide any functionality other than transporting data between client and server applications, it is the backend server implementation which provides the Publish and Subscribe capability. For this reason it is essential to think about the sub-protocol your applications will use when communicating over web-sockets, without this you will have applications sending garbage to one another and not achieving anything useful.

Considerations for Architects and Developers

Web Sockets provide the core communications capability for realtime data transfer, they handle the establishment of the connection and transfer of data. From a client-server application perspective there are a number of challenges that Architects and Developers need to be aware of when designing and building solutions around Web Sockets.

Connection Management

Unlike stateless HTTP, WebSocket connections create long-lived connections that must be actively managed. Networks are often unreliable with interruptions causing connections to drop. When you build with Web Sockets both the client and server developers must be mindful of the connections and their state - the management required is dependent on the actual features and function of the application sub-protocol being implemented. You will need to consider the following for managing connections:

Implementing reconnection logic with exponential backoff on both client and server
Design for graceful handling of unexpected disconnections.
Consider connection monitoring with Ping/Pong frames to detect "zombie" connections.
Ensure both client and server can detect "half-open" connections where TCP remains connected but the application level communication has failed.
Handle abrupt disconnect from clients which can occur through application crashes or browser tab closure or refresh scenarios to ensure you have clean disconnection logic.

Message Ordering and Delivery Guarantees

Web Sockets guarantee in-order delivery on a single connection but lack mechanisms for handling reconnection scenarios. When clients are reconnecting you need to consider the following:

Implement application-level sequence numbers for all messages to detect missed updates.
Build acknowledgment protocols for critical messages with timeout-based retransmission.
Design idempotency into message handling to safely process potential duplicates during reconnection.
Consider queuing unacknowledged messages for retransmission after re-establishing the Web Socket connection.

State Synchronization after Disruption

Web Sockets do not provide delivery guarantees or a way to rebuild state following a disconnection. When connections drop, there is no built-in way to determine what state was lost. You will need to consider designing a mechanism to deal with these scenarios:

Design efficient delta synchronization protocols to minimize data transfer after reconnection.
Consider event sourcing approaches where clients can request specific missed events.
Build "catchup" mechanisms that determine precisely what updates a reconnecting client needs.

Resource Management and Throttling

Web Sockets lack built-in flow control beyond TCP's mechanisms. This places the burden of resource management and throttling on you as a service provider. You need to consider the following:

Implement application-level back pressure signals when receivers can't keep up with message volume
Design rate-limiting systems that account for both message frequency and payload size
Consider prioritization schemes for different message types during high-load scenarios
Monitor per-connection memory usage, especially when handling large message fragments
Implement circuit-breakers that degrade service gracefully under extreme load

Frame Size and Message Fragmentation

Large messages create special challenges with Web Sockets. Web Sockets typically handle a smaller message frame size (e.g. 32kb), which means larger messages are broken up into smaller chunks which need to be reconstituted on the receiving end. You will need to:

Design protocols that properly handle multi-frame messages, especially during connection instability
Implement timeout mechanisms for partially received fragmented messages.
Consider streaming approaches for large data transfers instead of single large messages.
Test fragmentation edge cases including control frames interleaved with fragmented messages.

Standardized Keep-Alive and Health Monitoring

Since Web Sockets are a bi-directional communication channel, both client and server are responsible for detecting the connection stability and health. Your client and server applications will need to consider some or all of the following:

Establish consistent Ping/Pong usage patterns across your system with clear timing parameters.
Implement application-level heartbeats separate from WebSocket protocol mechanisms.
Design escalating health check procedures that can distinguish between different failure modes.
Consider how firewalls and proxies may affect long-lived connections with timeout policies.

Request-Response Correlation

Web Sockets provide a message stream with no built-in request-response syntax. These are left to the underlying sub-protocol to design for and consider. In order to ensure your protocol and messaging design is robust you should consider:

Implement message correlation IDs to match responses with their originating requests.
Design timeout and retry mechanisms for requests that don't receive timely responses.
Consider how concurrent operations will be managed without native request multiplexing.

Security Implications

Long-lived connections introduce unique security concerns which must be considered in your design. Think about some or all of the following:

Implement periodic re-authentication without disrupting connections - tokens expire and you need to refresh them while keeping the connection alive.
Consider rate limiting to prevent DoS attacks - both connection attempts and message frequency need monitoring.
Define clear message validation rules in your sub-protocol - validate all incoming messages before processing them.
Use secure WebSocket connections (wss://) in production to protect data in transit.
Verify the Origin header to prevent cross-site WebSocket hijacking attempts.
Implement timeouts for inactive connections to free server resources.

Testing Strategy

Web Sockets require more complex testing approaches due to the challenges already discussed. Think about and consider the following testing strategies to validate your solution:

Simulate connection failures and recoveries to ensure your designs function as expected, particularly consider the failure scenarios which is where problems and edge cases will occur.
Test partial message delivery and fragmentation to ensure your application is resilient to network interruptions. Web Socket data occurs in frames which are smaller in size to the overall message - this adds to the testing complexities.
Verify correct behavior under high message volumes to ensure your sub-protocol design functions correctly.

Protocol Design

The sub-protocol you design will impact on the functioning and reliability of your application. When thinking about the sub-protocol design you need to consider:

Message acknowledgment mechanisms for critical updates. If you need to guarantee a message delivery - this needs to be a specific design choice in your architecture.
Define clear message structures with versioning support so that you can support backwards compatibility.
Consider binary formats (Protocol Buffers, MessagePack) for high-throughput applications.

Designing messaging protocols is not easy, and must be a carefully considered part of your solution design. The sub-protocol should consider many of the elements we have discussed throughout this section, some of them are protocol specific which define the semantics of how the client and server communicate.

Remember Web Sockets don't Pub-Sub!

Web Sockets provide the communication channel, the responsibility for building reliable, scalable systems on top of this communication channel requires significant additional design and development effort. In my opinion success or failure of your application design comes down to the underlying sub-protocol. When implementing Web Sockets as your underlying communication mechanism make sure that you fully consider the impact of HOW your systems will talk - this will determine whether you succeed or fail because Web Sockets don't do pub-sub, your client and server applications do that - Web Sockets is your data pipe between them and everything else you have to build or buy!

When Serverless Goes Wrong!!?!?

Michael Walmsley — Wed, 26 Apr 2023 14:04:35 +0000

Recently, I spoke at two meetups on ”Unlocking Serverless Observability”, which is an important topic, so I have decided to publish an article about the demo project I put together to help illustrate - When Serverless Goes Wrong.

A Working Sample

Supporting my recent talks and this article is a working (and non-working) example of an Event Driven data integration Architecture that can be found on my GitHub page. The README is comprehensive and explains how to install and run the performance tests if you want to see it yourself.

WARNING: Executing a performance test even with AWS Lambda may incur charges on your AWS Account. Given the generous free tier, this is unlikely, but you have been warned.

About the Project

Alice works for Widgets Incorporated, a popular online retailer of technology widgets. Alice keeps the lights on for the e-commerce website running on a large EC2 Instance on the AWS cloud. Alice’s most important job is ensuring the weekly sales export runs for the sales and marketing team.

One day, Max from sales and marketing seeks out Alice and demands something to be done about making sales data available more than just weekly! In the same conversation, he also shares that they are changing delivery providers and they will need to integrate into Ace Courier’s scheduling API to coordinate deliveries.

Alice is very worried about adding more processing to the existing e-commerce platform. It is at capacity, and Widgets Inc. doesn’t have a large budget to do much more about it! Alice digs through documentation wikis and stack overflow, trying to think how this problem can be resolved. When she was ready to give up, someone replied to her question about stack overflow. Widgets Inc.’s e-commerce solution has a lesser-known notification webhook feature that can be turned on through configuration.

Excitedly she turns to Serverless Land to learn more about Event Driven Architecture so she can build a serverless event hub to handle live order notifications. With this simple configuration change, she knows she can deliver live sales data for Max in Sales and Marketing and do a live integration with Ace Courier’s scheduling API.

On Serverless Land, Alice finds a GitHub reference to an Event Driven pattern which will be useful to solve her exact problem. The pattern she found onServerless Land is for Java, but that is okay because the existing sam template will accelerate the IaC aspect of building out the serverless event hub in Python.

Alice has been learning about Serverless with Python and has been reading about SAM CLI and AWS Lambda Powertools for Python.

How Alice Built it?

Alice used AWS Lambda power tools for logging and tracing and simplifying her code. For the Delivery API Lambda, she used the tenacity library, which is great for adding retries to any function in Python to ensure the lambda recovered from any failures.

Alice used AWS Lambda Powertools for Python which has an excellent structured logger. She made good use of this and set the service name to enable logs to be grouped by each service processing data. Alice leveraged the correlation_id_path log setting, which enables json-path extraction of attribute values from the AWS Event being processed to automatically populate the correlation_id so she could trace transactions across the entire distributed stack. Alice also listened to my Getting Started tips by making sure the logs tell a story and using the START, COMPLETE and FAILED status values as a status JOSN attribute for key logs to assist in tracing.

Alice did quite well and completed the build reasonably quickly - she had good logging and retries for API integration failures which are all great considerations for building a resilient event-driven architecture.

So What went Wrong?

One of the key things Alice didn’t consider in her build, which is common for many teams I work with, is understanding the transaction Volumes that the architecture will be processing. This is critical to understand since Serverless is hyper-scalable and will handle almost any traffic you throw at it! Alice also didn’t consider the potential limits of the external Courier API she was interacting with - understanding the constraint of ALL your downstream systems is just as critical! If you know your volumes and downstream limits - you will quickly learn up-front at design time whether any scalability boundaries will be problematic.

When Serverless Goes Wrong

To demonstrate the problem when Scalability Boundaries clash, I built Alice’s demo architecture and introduced a deliberate Scalability boundary through a slow-api service representing Ace Couriers.

The Demo Project Constraints

There are 3 components to this project:

NotificationFunction - this takes the payload from the API, injects a correlation_id and forwards it to EventBridge as a notify-order event.
DeliveryFunction - This is a simple function that takes the detail of the event, removes the meta-data from the body and sends this to the configured API endpoint. This lambda uses the tenacity library for retries in the code with a wait time between 3 and 8 seconds on any failure from calling the “slow api”.
SlowAPI - This is an Api that has been setup to mimic an unstable real-world API and has been setup with the following constraints:
- 20% of the API Calls will fail immediately.
- The API will take between 0 and 2 seconds to return a response when it runs successfully.

The SlowApi lambda is triggered by an API Gateway route with an API Key and usage plan set to restrict the number of transactions through it to 10 per second: BurstLimit = 10, RateLimit = 10. The NotificationHandler is also triggered by an API Gateway with no API Key and no rate limiting.

There is a configured artillery test file that can be used to run a performance test with the following characteristics:

For 1 minute test will hit the Notification API at approx. 2 transactions per second
For the next 2 minutes, transactions will ramp up to 5 transactions per second
For the next 10 minutes, transactions will ramp from 4 per second to 15 per second and then remain at this level which is a level that exceeds the SlowAPI transaction limit.

There are 2 branches on the demo project - no-flow-control is what Alice built. flow-control is the same architecture but with flow control introduced to ensure the event integration flow is controlled.

What Happened During the Performance Test?

The Slow API worked quite well - it had errors, but these were planned errors, approximately 20% of all requests. So the graph below is a fairly solid representation of a 20% error rate with the green line wavering around the 80% success mark as planned. Nice to see part of the system worked!

As you can see from the Delivery Lambda Success dashboard graph below, the Delivery Lambda was fine for the first seven (7) minutes before the success rate started to drop. What is impressive from a Lambda perspective is that at the 8-minute mark, we have a success rate greater than 90%, whilst we have errors sitting at nearly 50 in that minute. This speaks to the automatic resilience that is built into the AWS Lambda asynchronous invocation mechanism. Sadly as time continues, the success rate bottoms out at approximately 20% at the 17-minute mark with a peak error rate of 1535 failed lambda invocations. This graph picture is something you will see a LOT on the metrics dashboard for your lambda functions when errors are occurring. The important thing to look at is the bottom number of the range for the % Success (where it bottoms out) since it may bottom out at 97%, which from an overall system health perspective, is still quite healthy.

When you have scalability boundaries clashing, what you will see is a steady growth in your overall Lambda Concurrency metric, as shown below. This graph peaks at 788, just shy of my account level limit of 1,000 concurrent invocations. A common reaction as people see this is to urgently reach out to your AWS Technical Account Manager and very quickly increase the account concurrency so that you have more headroom. Unfortunately, when you have this reaction, you only create a larger pool of concurrency for the out-of-control lambda function to draw from, which will make the graph peak much larger and effectively make your bill slightly larger.

When you have no flow control built into your system, there is nothing much you can do at this point but ride out the storm or deactivate lambda function triggers to stop your system from creating bigger problems for the downstream systems. This is the key architectural component missing for this event-driven architecture - Flow Control. The overall invocation count metric graph tells a very similar story, as expected.

Why Did this happen?

The scale madness happens because we have a Scalability Boundary between the Delivery API Lambda and the Ace Courier System. Anytime we have a hyper-scalable system talking to a less scalable system or a system of unknown scale, this is a Scalability Boundary. This is where scale differences occur and cause problems. As Serverless Architects, we need to be able to identify these boundaries and design our architectures to ensure our system remains resilient and working at all times. This boundary can also be considered a domain boundary. We must know all interactions beyond our bounded context and manage failure carefully to avoid losing data due to a Retry Storm, which is what causes the graph peaks and high concurrency load.

How to fix

To resolve scalability boundary clashes, we need to add a flow control buffer that can provide a retry mechanism on failure. The best example is using an SQS queue as the buffer element coupled with maximum concurrency configuration on the Lambda trigger’s Event Source Mapping. This combination is the most powerful flow control mechanism available on AWS today.

How does Flow Control help?

The slow API in our system with flow control is behaving as expected. The success rate is wavering around the 80% mark by design. The error rates are much lower - this is because of the introduction of flow control, which controls the number of concurrent calls into the Slow API endpoint. As you can see, the Slow API (or Ace Couriers System) is much calmer and not spiralling out of control like before. A graph like this is not ideal because there are errors, but it’s not showing a system that is completely out of control, so there is time to work to correct the problem. In this case, the error rate is because we coded it in, but in a real system with transient errors like this we need to look at logging to determine the issue and then deploy a fix.

The Delivery API Success Graph shows the switching of success to error and back again - but notice the lowest value on the Y-axis is around the 98% mark. So although there are errors, it is not disastrous (yet).

With flow control in place, the concurrent lambda execution is way different - it is a very steady flat line which is what we expect to see in a healthy system. It is hovering around a total concurrency of 20 across the entire system, which is very healthy!

Invocations are also okay, as you can see from the graph.

In these last 2 graphs, you can see 2 anomalous spikes. Something weird happened with the metrics for the Notification function. At 13:55, cloud watch stopped recording metrics for this function and instead recorded all counts in 2 batches at 14:12 and again at 14:28. Outside of these 2 anomalies, you can notice the steady, even pace at which the lambda invocations happen through the flow control mechanism provided by the SQS Queue.

Why is Structured Logging so Important

One of the great things Alice did well was to use AWS Lambda Powertools for the Logging utility, which emitted structured logs in JSON structure. Part of the logging utility enables capturing of the transaction correlation_id along with function details from the lambda context and a defined service name, like the entry shown below.

The additional fields enable Alice to search for everything that happened to a single transaction to Tell the Story of what happened. An example output of a correlation_id story is shown below:

A feature I get all my teams to do is to emit a START, COMPLETED, and FAILED log message as a status. This is powerful to query logs over a timeframe to determine whether everything was processed correctly or not. Often this message is built into the Lambda handler framework itself through middleware components so that developers don’t have to do anything to get this happening. In this way we can compare the counts of START messages in the notify-handler service to the COMPLETED status messages in the delivery-handler service.

As you can see, for the No Flow Control test - the actual success rate was terrible.

Where the Flow Control test had a successful outcome.

Conclusion

As you can see from this experiment, adding Flow Control between external communication points within each bounded context of your distributed system makes for a more successful event processing flow. The other points that come to mind that are critical for building Event-Driven Architecture for success are:

Understand your expected Volumes up-front and model them. Understand what your real Transaction rate per second (TPS) is.
identify the rate limits and constraints of systems you interface to. It’s important to understand where the scale clashes will occur.
Add flow control to your Bounded Domains to assist with scalability boundary clashes.
Use structured logging to enable forensic analysis when things go wrong.
Emit some simple messages to highlight the beginning of lambda execution and another to highlight whether it was successful. Use these to create metrics for comparison and to measure the success of your system.
Think about Business KPI metrics - I keep saying this but do think about them - they are also important, and the absence of a business metric can indicate something not working correctly.

Observability is important, and it’s not that hard to get it right. With some planning and the useful techniques highlighted here, you can quickly measure your serverless system’s success!

Introducing Serverless DNA - A New Site dedicated to Serverless!

Michael Walmsley — Sun, 29 Jan 2023 01:13:29 +0000

One thing I have learned over the past year while writing blog articles on Serverless to help people learn is that it is not straightforward, and the skills you need are more than just writing software or deploying infrastructure. You need to understand a broad range of topics to design and build solutions with Serverless successfully. This makes blog articles that go deep really long and difficult to read and understand because there is so much to cover!

To be successful with Serverless, you need to have some understanding of the following:

Infrastructure as Code
Security
Event Driven Architectures
Available Managed Services
Code Frameworks
Software Engineering
and the List goes on ... !

So I have created a new blog site for all my future writing based around smaller, focused articles that are interlinked and weave together to create a complete picture of each topic. The articles are organised into focused groups and will be linked so people can browse through each topic and learn information in smaller doses which will be easier to read, understand and share.

What is Serverless DNA?

The "DNA" stands for "Digital Native Architecture", which is how I see Serverless fitting into the broad world of cloud technology.

Serverless DNA takes the form of Small Blog articles, "Mini-blogs" to create knowledge about a specific Strand of Serverless Technology. Each Strand of knowledge may be linked to one or more other Strands which are related and should also be looked at to help expand and grow your knowledge of Serverless.

This first release on serverlessdna.com is the MVP of the DNA concept for Serverless knowledge - I want to grow this into more than just my blog site but also enable it as a community resource linking in content from around the globe from other contributors to help categories, link and collate the body of Serverless knowledge out there today. Find out more by browsing the About Strand - which is where I will be sharing the public roadmap for future development.

I hope you find this new site useful and I really look forward to your thoughts and comments on this new beginning for my writing in 2023.

Being an AWS Community Builder in 2022

Michael Walmsley — Tue, 20 Dec 2022 22:17:54 +0000

What surprises you most about the AWS Community builders program?

I am always amazed at how "ON" this community is. There is always something happening and always someone online somewhere to provide you with unconditional support in answering questions. Everybody here is so caring and willing to share and really just wants to help you to be better at building!

It's a fabulous AWS Community and the giving feeling here is completely infectious and addictive!

What is your background and your experience with AWS?

I have been working as an application developer and architect for a while now - I keep telling people I witnessed the birth of the internet as we know it today when the Mosaic browser popularised the World Wide Web (WWW). Back then, you spent time building your own software frameworks as there was nothing around and open source was only just starting - so when I look back my entire career has been about searching for that perfect software architecture, and working towards providing mechanisms for delivering the perfect digital experience.

Rolling forward a lot of years I started working with AWS in 2015 working for smaller digital agencies. I worked to build out solutions using an ecosystem of docker containers which were deployed on EC2 instances. At this time API gateway was released and I started exploring this "Serverless" thing that had started, and really quickly I started to get hooked!

I really got to know AWS Serverless deeply around 2018 where I had the opportunity to re-develop and improve the scoring backend for the 2019 Australian Open Digital Experience on ausopen.com. I designed and built a series of de-coupled, event driven microservices for delivering live scoring updates, stats, and schedules to the mobile and web front-ends - it was a wild ride but what you could get done in a short amount of time was astounding - it was the most immersive 12 weeks of my career and I see this as a real turning point to going all-in on cloud and Serverless.

My current claim to fame - "Ever watched the scores on the AusOpen app change? I did that, that was me, and the tennis ball that indicates who is serving - I wrote that code - I make it move".

In the past 2 years I have designed and built a number of Event Driven platforms using Serverless and each time I have built a new team and trained them in the Ways of Serverless and continually work to develop and accelerate serverless teams.

What is the biggest benefit you see from the program?

For me its the collaboration and community connections the AWS Community Builder program provides coupled with Webinars from AWS service teams and technical experts. Every webinar is a growth opportunity, being in this program is a growth opportunity - The amount of knowledge on AWS in one place is astounding and its there for immediate consumption - You just need to be sure you get some sleep and look after yourself!

What is the next swag item that you would like to get?

I always joke about loving swag and I think it's true - I LOVE swag! Jason Dunn and the team do a fabulous job of creating Swag items that are actually useful! I love and use all the swag I have been gifted so far.

Personally I have been eyeing off the "Jacket of Epic Builder Endurance" for a while now (Just saying!).

What are you eating for dinner today?

Tonight its Chicken Cacciatore - click the link for Jamie Oliver's recipe - I don't really have a recipe to be honest. For me its a whatever I happen to have kind of recipe, but Jamie's is pretty close!

Is there anything else you would like to share about the AWS Community Builder program in 2022?

I am incredibly thankful for being accepted into this community and am amazed at how caring the AWS Community leads are and the lengths they have gone to in helping us learn and be continually engaged.

I can also honestly say that my journey to re:Invent in 2022 was an incredible experience amplified by being a part of this community. Being looked after by our incredible AWS Community hosts who went out of their way to ensure we engaged with the service teams and had everything we needed, making re:Invent super special, THANKYOU!

Serverless Testing, Part 2: SOLID Architecture for Lambda

Michael Walmsley — Sun, 10 Apr 2022 08:45:41 +0000

Serverless Testing, Part 1: What I forgot at the beginning describes the start of my serverless story where I forget about plain old Software Engineering Principles, and I re-introduced the Dependency Inversion Principle, which is essential when writing software! I want to explore software architecture for serverless functions in this second instalment. You will learn about SOLID principles and we will break down Hexagonal Architectures and how these ideas work together to simplify building Serverless code.

In my experience, many teams who are moving into Serverless development are from a heavy infrastructure background and are less experienced in application development using software architecture principles. So I like to start by going back to basics and talking about SOLID principles of building software.

Single Responsibility Principle

A function should have one reason to change

Functions should have a single responsibility which is the only reason it should change. The more responsibilities your code has, the more reasons you will have to change it, and the more complex your functions will become over time. Keeping functions simple has several key advantages - it lowers code maintenance costs and makes them faster to execute. However, keeping functions simple is not as easy as it sounds and is part of the distributed design that causes friction in some teams who feel that serverless systems are complex to build or have too many moving parts. This aspect of serverless is the hardest to deal with and needs a disciplined approach to keeping your solutions simple.

Open / Closed Principle

Software entities (classes, modules, functions, etc.) should be open for extension but closed for modification.

This software principle means that classes should be open for extension but closed for modification meaning the inner working of a class or function should not need changing when a dependent object or class changes. You can achieve this through interfaces that extend your code's inner working by introducing a new implementation. Using interfaces in this way enhances the loose coupling of dependencies.

I also like to apply this to serverless functions when you need to write code. I always try and make my functions data-driven and configurable so that the processing relies on only the event triggering the function. This principle applied to serverless is about not changing your original service to deal with specific behaviour in a specialised context. Instead, look to make these exceptional cases an option or feature flag and encourage composability to add additional processing.

Liskov's Substitution Principle

Overridden class methods must have the same input parameters as their superclass. So, if it looks like a duck and behaves like a duck, it is a duck, and if you want a new duck, make sure it quacks the same as all the others.

Liskov's principle states that if you use a subclass in place of its parent class anywhere in your software, nothing will break. In simple terms, parent classes define a contract that sub-classes should never alter - violating this principle means clients who try to use this sub-class will no longer work as expected without being modified.

In serverless, a new version of your service should always be able to replace a previous version without breaking anything. This also puts our design focus on service consumers or customers who do not need to change their code when a new version is deployed, which would be considered a breaking change for the Interface.

Interface Segregation Principle

No code should be forced to depend on methods it does not use

The Interface Segregation principle keeps your defined interfaces simple and specific to their purpose. Keeping interfaces clean is essential; otherwise, every implementation is forced to implement functions rarely used, which is a waste of effort and increases software complexity.

This principle also helps in serverless designs by ensuring that we design simple, single-purpose interfaces for our services. A service that does too much will expose too many details and may leak internal implementation details to consumers, which is always a mistake for any microservice architecture. Keeping interfaces simple and to the point is what this principle is about, and we should be strong in applying it to every service boundary we design.

Dependency Inversion Principle

High-level modules should not depend on low-level modules. Both should depend on abstractions.

Abstractions should not depend on details. Details should depend on abstractions.

Dependency Inversion is all about creating functional abstractions between different layers of our code. This principle enforces the use of interfaces for every dependency, enabling the simple replacement of any component in our code quickly and easily. To embrace this principle means when designing an interaction between a high-level module and a lower level, one should be created with the interactions in mind, which will dictate the overall interface design. In this way, the coupling of our software components is reduced, and we can introduce new implementations at any time, which is ideal when we have embraced all the SOLID principles explained here.

The serverless use-case for this principle is really in building out strong software domain abstractions around cloud services to isolate business logic from the complexity of the cloud services it uses. This was the main point I introduced in Part 1 of this series.

Hexagonal Software Architecture

Hexagonal Architecture is all the rage in the Serverless world and has been written about by many people:

Ready for changes with Hexagonal Architecture on the Netflix TechBlog by Damir Svrtan and Sergii Makagon

Developing evolutionary architecture with AWS Lambda on the AWS Blog by Luca Mezzalira

Service Ports: Finding a Loosely Coupled Utopia with Event-Driven Serverless on Serverless Transformations by Ben Ellerby

These articles describe the benefits and tell you why you should be using hexagonal architecture. They all talk about Ports and Adapters as the magic you need to isolate your business domain logic from the more specific lower-level code that deals with writing data to databases, file systems or API interfaces. I don't like to use "hexagonal" or "ports". Instead, I prefer to talk about software patterns.

Building your software using this concept is nothing more than embracing the Dependency Inversion Principle from the SOLID principles we have already discussed. In hexagonal architectures, Ports are nothing more than well-defined Interfaces clearly defining your business domain interactions with external dependencies. Adapters are specific implementations of these business domain interfaces (Ports) to enable the de-coupling of your business domain logic from the cloud environment you deploy to. Adding dependency injection into the mix at this point is the real magic; Enabling injection of Adapters into your business domain logic allows a lot of freedom in building and testing your code.

I strongly feel there is a lot of effort focused on developers mocking SDK elements of AWS and other cloud providers. A lot of time is wasted trying to work out how to mock SDK responses from SDK API calls. By adopting business domain Interfaces and implementing them with Adapters, you can instantly create mocks using a mock implementation of the Interface for local testing purposes. This means your business domain developers do not need to know about the AWS SDK dependencies. This can be locked away within the Adapter implementations, which can be handled by framework developers who can be specialists for your cloud environment. Combining Adapters with simple dependency injection allows your adapter dependencies to be injected, meaning business domain code does not need to change whether you are unit testing, integration testing, or deploying to the cloud. You will have adapters for each of these environments and can drive the injection of these dependencies when needed so all your business domain logic can be quickly and simply unit tested.

A huge benefit of adopting a Hexagonal architecture is to isolate your business logic from your cloud and serverless components that make up your application. This means you can now have some separation of responsibilities within your team - Experts on your cloud integration and experts on your business domain logic. With this isolation, you can have less cloud experienced programmers focus on business logic and more deeply cloud experienced developers on Cloud integration - this separation of concerns is powerful.

The Road to Hexagonal Applications - Service Oriented Architecture (SOA)

Clean hexagonal architecture comes down to software engineering principles we have discussed throughout this article, and I will throw in one more term - Service Oriented Architecture (SOA). Wrapping our business domain logic into Service classes creates a complete separation between your cloud environment and application logic; it also becomes a natural break between cloud expert developers and business domain developers - something I have been searching for to create layered teams with specific skill-sets across cloud and the business domain.

Using Services as the core base also allows direct and straightforward dependency injection of the various dependent interface adapters (Ports) required for the service to function. The following code snippet shows an example of a base abstract style python class for a Service Interface (Port). It accepts several other adapters for core functionality required, including Logging, Notifications and Storage. Through dependency injection for domain adapters - the business domain logic is now wholly isolated via the Dependency Inversion Principle I mentioned earlier.

Adding Business Functional Adapters (Ports)

In the Service example, several adapters are injected as dependencies. The following code sample shows how you can structure the notification adapter (Port), which has a business-focused method notify_new_user that is functional. In this way, we separate the detail of the notification implementation from our domain programmers. This also means we can pivot the actual implementation without changing business domain code - so from a SOLID perspective, we are now embracing the Open-Closed Principle and the Dependency Inversion Principle - two powerful software engineering techniques.

Bringing it all together

The following example shows how an AWS Lambda handler would apply the techniques discussed in this article. An adapter class (UserEventMapper) translates the Lambda event into a User model, which can be passed into the Service instance to process the new_user. The Service response is also mapped by an adapter class (NewUserResponse) to enable an actual AWS Lambda Service response based on the service used to trigger this function. This creates a very clear Lambda function that anyone can look at and understand what is happening. The adapter classes used can have their implementations changed to alter how this function is being triggered—all without making changes to the business domain logic.

We have explored the core of software engineering - SOLID principles, Dependency Inversion, Dependency Injection and Service-Oriented Architecture (SOA). Combining these techniques can assist in simplifying your development team's journey to Serverless by abstracting away the cloud complexity that Serverless developers are traditionally confronted with. Using these techniques discussed here, we can reduce the cognitive load developers have to contend with in creating serverless solutions. This all does feel like it goes against the simplicity of serverless, but we must acknowledge that we are mostly building software solutions at the end of the day. I say mostly as we should be embracing serverless services where possible, but like Jared Short says - "When you write Serverless code, you need to own it", so engineer it properly and take control.

Own it is what I say. Own it with SOLID principles, Interfaces and Adapters and Service-Oriented Architecture.

Serverless Testing, Part 1: What I forgot at the beginning

Michael Walmsley — Mon, 10 Jan 2022 12:07:04 +0000

This is the first in a series of articles about Serverless software architecture and testing. I am collating a summary of what I have learned on my journey to Serverless and am starting with this introduction, highlighting my thoughts on what I forgot when I started learning about serverless. This series will be made up of three parts:

Part 1: What I forgot at the beginning

Part 2: SOLID Architecture for Lambda

Part 3: Simplify Testing with SOLID Architecture

Serverless Testing has been a hot topic in recent times. Many of the articles focus on general approaches, tooling to automate the creation of temporary stacks, and tell us to stop emulating the cloud locally and test in the cloud! In this article, I won't be covering these tools and approaches. Instead, I want to focus on what I forgot on my Serverless journey, and hopefully, it will help you not make the same mistake.

I started my journey with Serverless over four years ago when Lambda was relatively new, and we were all trying to work out just how this new cloud technology worked, scaled and could be used to create real, working, resilient solutions. There are many things to learn about at first - Lambda, SQS, SNS, DynamoDB, and the list goes on. So many new things that we all get wrapped up in learning how these products work.

When I started building serverless, I started with simple code; they were just functions, after all. Then, I quickly moved to local emulation of SQS, SNS, etc., to do what we have always been encouraged as software engineers - do end-to-end local testing. With mixed success, I moved back to unit testing with event fixtures to test my core business logic and continued to refine my testing knowledge and skills. There are always complexities involved with testing lambda code; There are interactions with cloud managed services such as S3, SNS, SQS or DynamoDB, which have AWS SDK clients to manipulate data being stored or processed. Now I spent a lot of time working out how to mock these AWS dependencies in my unit tests and make sure the SDK calls expected were made. Every new service seemed to introduce a new challenge; I spent more time writing mocks and getting unit tests working in this foreign land of AWS dependencies than building solutions and started to feel less productive. I have spent a lot of time the past few months reflecting on my journey since it began and how I have changed my thinking in many areas, particularly around testing serverless solutions.

I strongly feel I got wrapped up in the details of all the new services I had to deal with and coming to terms with how they all worked. The cognitive load on putting this all together at first is a lot! Combining this additional learning and the understanding that we are just writing functions means we expect to be writing fewer lines and simpler code. I was doing just this when I first started and looking back. I can see what I forgot about! I forgot all about plain old Software Engineering!

As a professional Software Engineer with a lot of experience, this is quite a confronting revelation. However, looking at conversations within the community over the past year, I feel I am not alone here. So what do I mean when I say I forgot about plain old Software Engineering? Three small words come to mind - Dependency Inversion Principle.

Dependency Inversion Principle

This core principle of Object-Oriented design sounds scarier and more confusing than it is. The following describes this design principle defined by Robert C. Martin:

High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).
Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

Dependency Inversion is all about creating functional abstractions between different layers of our code. In a Serverless function context, this technique allows us to focus on more meaningful, descriptive functions to perform actions for our code rather than getting bogged down in the detail of an AWS SDK client for DynamoDB and its associated API calls. Creating these abstractions also removes the implementation detail and complexity of the cloud service mocking of the past. It allows quick and clean testing of each Interface since we can introduce a test stand-in implementation for any abstraction.

Class Diagram - Dependency inversion applied to Lambda handler

So what we need to look at is creating functional abstractions for any call from our core business logic out to the cloud. For example, in the diagram above, you can see the EventHandler depends on the DomainServiceInterface and the DomainLogicImplementation, which implements the DomainServiceInterface, depends on the StorageInterface. This allows the replacement of any green implementation blocks with Testing Stubs enabling all the components to be tested without creating complex mocks and test assertions. This simple software engineering principle is what I forgot about during the start of my Serverless journey. With the state of confusion and commentary around Serverless testing recently, I felt compelled to share my thoughts and what I forgot about when I first started. Hopefully, this will help you find your way quicker in this often-confusing space.

This article is the first in a new series I am writing on Serverless Software Architecture and Testing. These concepts go hand in hand as good architecture and code organisation naturally lead to better testing. My next article will be about SOLID Architectures for Lambda, taking the Dependency Inversion principle mentioned above further and deeper.

AWS Lambda Powertools: Idempotency, A Deeper Dive

Michael Walmsley — Sat, 23 Oct 2021 22:50:48 +0000

In an earlier article, I wrote about Making all your APIs Idempotent. I shared why idempotency is essential and introduced the AWS Lambda Powertools for Python, which provides a utility for adding idempotency to your Python built lambdas in a quick and easy to use way. My article focused on an API as the reason for requiring idempotency and, while accurate, it is not the only use-case for the Powertools Idempotency utility. In this article, I dive deeper into the Powertools utility and highlight more features and use cases.

How Idempotency Works

The idempotency utility is quite a complex feature wrapped up into a simple package to make it easy and approachable to use. The idempotent utility follows these steps:

Calculate a unique idempotent token using the request payload and check the idempotent store to see if the transaction has been completed or is currently in flight.
If completed, the utility will return the stored result from the first execution; If in-flight, the utility will raise an IdempotencyAlreadyInProgressError exception signalling to the caller it is safe to retry the operation.
If no transaction exists, the utility will save an in-flight record into the store and call the function to process the transaction. When the function completes, the utility will keep a copy of the response so that future invocations can receive the same result.

At its core, the idempotent implementation for power tools derives a hash value (md5 by default) for the idempotent token of the transaction. Therefore, calculating this unique token is the most critical part that we need to understand to ensure idempotency works correctly.

Calculating the Idempotent Token

AWS Powertools has your back in this department and will take on board sane, consistent defaults that will work in more straightforward use cases by calculating the idempotent token using the entire message body. However, if your use case is not simple, you must understand how your data changes between invocations. When processing an AWS Lambda event, many of the fields within the event will contain data that changes between each request, for example, "requestId" and "requestTime" in the AWS API gateway Proxy event, which vary for each API invocation.

For non-default hash keys, you will want to use the IdempotencyConfig object and ensure you set the event_key_jmespath string to allow powertools to extract the unique idempotency token from the Lambda event for the hash key.
Understanding how JMESPath and the Powertools JMESPath powertools_json functions decode strings to JSON objects is essential for the idempotency utility.

Making a JSON REST API Idempotent

An excellent example is using the utility with API gateway proxy events for a JSON REST-based API. Unfortunately, the AWS API Gateway Event's body attribute is a string value that is not ideal when trying to assert the idempotency of your JSON Payload.
By definition, attributes in a JSON Payload for a REST API have no order associated with them, meaning the following examples are considered identical in terms of an API transaction:

{
    "transaction_id": "bd784218-31cc-46be-b66a-a9c38b9a2fe5",
    "user_id": "10000233220",
    "email": "me@email.com"
}

is the same as,

{
    "email": "me@email.com",
    "user_id": "10000233220",
    "transaction_id": "bd784218-31cc-46be-b66a-a9c38b9a2fe5"
}

The Powertools idempotency implementation considers this detail, but only if you ensure the body is decoded into a JSON object using the powertools_json built-in JMESPath function. To achieve this outcome for the above example, you would need to set up the IdempotencyConfig as follows:

config = IdempotencyConfig(event_key_jmespath="powertools_json(body)")

With the powertools_json built-in function, the data given to the Idempotent Utility's hash generation function will be transformed into a python Dictionary object, allowing the hash generator to convert the data into a sorted JSON serialized string. In this way, your API will remain idempotent regardless of the attribute ordering in the API JSON body.

When choosing the parts of your message payload for the unique idempotency key, it is critical to realize that a string value can contain whitespace and newlines, affecting the hash outcome.
I strongly recommend reviewing the JMESPath tutorial page to understand how the JMESPath utility works; it is a powerful tool in your python arsenal and is essential in ensuring the Powertools idempotency utility works for you correctly.

Idempotent Decorators for Everything!

The core of the idempotency utility are the python function decorators (yes, there is more than one!). A recent addition to AWS Lambda Powertools for Python is the idempotent_function decorator, which provides idempotency to any synchronous Python function. So now you can add idempotency to absolutely anything, not just Lambda handlers!

Making any function Idempotent

Using the standard idempotent_function decorator, you can tell Powertools how to calculate the idempotency key using any of the function parameters. The IdempotencyConfig is also available for customizing how the utility will work utilizing the event_key_jmespath as mentioned above.

The following code example showcases applying the idempotent_function decorator to the SQS batch processor function making your SQS queue processing completely safe from processing the same message multiple times.

Payload Validation

The AWS Builder's Library article on Making retries safe with Idempotent APIs covers the need for payload validation when data that is not part of the idempotent token changes between requests. You need to consider these scenarios carefully so you do not confuse your API consumers; Powertools has a payload validation feature you can leverage to cover this situation.

Testing Your Solution

Once you have designed and implemented idempotency in your solution you will want to test it thoroughly before making it live. You can test your solution using a local dynamo DB or mocks to save on cloud costs; Powertools covers code testing scenarios completely in their online documentation.

Conclusion

My everyday work is building and training Serverless teams and making sure they build reliably, with resilience and taking note of the AWS Well-Architected Principles documented in the Serverless Lens. I always recommend developing on AWS Lambda with the Python runtime since I can rely on teams meeting the Well-Architected principles through the AWS Lambda Powertools. For groups who insist on Typescript or .NET, I strongly urge them to use Python since it has Powertools, the swiss army knife that AWS Lambda developers need for every project!

If switching your development to Python for AWS Lambda Powertools is not for you, I recommend heading over to the AWS Powertools Roadmap and supporting the Typescript and .NET feature requests by adding a 👍.

Correlating Transactions with Zero Code

Michael Walmsley — Tue, 12 Oct 2021 13:10:34 +0000

I take pride in writing detailed, technical articles filled with solid code examples and links to Github so you can review and learn from my serverless journey. Today, I think this will be my shortest ever article, and it lacks all these things since starting on this Zero Code adventure.

There are a lot of articles out there on correlating transactions across distributed systems and plenty of technical frameworks and solutions for providing transaction traceability. However, correlating transactions as the data passes through our services is essential to observe our system behaviour. So, as developers, we reach for what we know best our development tools, software frameworks and technical solutions.

Adding Correlation Ids, tracing codes and segment identifiers takes effort and time.

Usually, we place these identifiers in HTTP Headers and other unique Attribute locations for Cloud-Native services so we can hide these away and provide them as transparent travellers across our distributed systems. In an AWS Managed service sense, this means we need to know how to pack and unpack these identifiers at each step through our serverless systems - across SQS boundaries, through EventBridge events and via SNS topics. These cloud-native services each have a different mechanism to save, transport, and unpack these keys to observability.

But what if there was a different way without all this complexity and code?

I want to share with you my thinking on this and what I have arrived at to resolve this technical problem in a way that transcends the AWS service complexities I just mentioned. Nowadays, I believe in using the language of integration, and for tracing transactions through distributed services, I create an internal integration language encapsulating these observability identifiers.

Here is an example of a message structure from a system I am working on right now:

{
    "correlation_id": "b58e7f98-2b58-11ec-8d3d-0242ac130003",
    "message_id": "bde284dc-2b58-11ec-8d3d-0242ac130003",
    "trace_id": "3965a72e-2b59-11ec-8d3d-0242ac130003",
    "data":
    {
        "id": "7e6c7638-218c-4947-8d84-a20a9cf68acd",
        "firstname": "michael",
        "lastname": "walmsley",
        "email": "michael@myemail.com"
    }
}

The data represents the original payload received from the initial caller of the service, usually via an API endpoint. The API entry point is responsible for validating the data to ensure it is correct before transforming it into the internal message format containing the tracing identifiers (correlation_id, message_id, tracing_id). Using a standard messaging structure like this means we no longer worry about changing how we interact with each of the Managed Services we are using since our correlation data is now directly embedded within our message data. All of our messages now have correlation identifiers that pass through data services like SQS, Eventbridge, SNS, Kafka, Kinesis, etc., without the need for any custom code to embed meta-data or other custom markers for correlating our transactions.

This simple approach of defining an internal service messaging format allows your transaction data and correlation data to always be visible within every component of your distributed Serverless Architecture. Furthermore, we achieve Transaction observability within our Serverless data processing systems using nothing more than our already prepared Cloudwatch Logging utilities and Zero additional code!

Serverless Integration, Zero Code

Michael Walmsley — Sat, 09 Oct 2021 10:03:40 +0000

The words of Farah Campbell and Ben Kehoe on "The Serverless Mindset" has inspired me to level up my thinking and approach to Serverless integration.

So let me share my journey to zero code integration with you.

For my first foray into zero code integration, I decided to keep things simple and play with the classic integration pattern of sending data via an API into an SQS queue for downstream processing. Most serverless websites will have you build an architecture like this, an API gateway triggering a Lambda that pushes to the SQS queue for downstream processing.

This pattern is easy to build, deploy and get working with any of the main Serverless frameworks; my weapon of choice is the Serverless Framework from Serverless Inc. The Serverless Framework focuses on deploying Lambda functions and makes this task quick and easy. For example, to build the Lambda processor pushing to SQS looks something like this.


service: api-lambda-sqs

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

functions:
  apiPushSQS:
    handler: handler.api_push_sqs
    events:
      - http:
          method: POST
          path: /
    environment:
      THE_QUEUE: !Ref downstreamSQS

resources:
  Resources:

    downstreamSQS:
      Type: AWS::SQS::Queue

Building out the pattern is simple, and the framework focuses on Lambda functions and their events which makes building and deploying quick and easy. I managed to do this in minutes and felt productive; the instant win had my endorphins flowing, and I was conquering it!

But is this the ideal serverless solution?

I am sure Ben Kehoe would say, "If you came along to my talk today to hear about Lambda, you are in the wrong place", and these words are echoing in my head right now as I write this and think how I can do this without code.

We all know the AWS API Gateway supports direct, native integrations with many services, and I have designed many systems around these patterns. Still, I have never actually built one using the Serverless framework; until now.

To achieve direct integrations from API gateway to other Services involves understanding how to configure the Integration request, and there are choices as to how we can go about this. I started by configuring the API via the console and followed one of the few blog articles I could find on this topic to set it up, which you can look at here.

This article allowed me to see the steps I needed to complete:

Create the SQS Queue
Create the IAM Policy allowing API Gateway to use the sqs:SendMessage action on my SQS Queue.
Create the role to attach the policy to and enable API gateway to assume the role.
Create the API and define a POST method that we can use to map the incoming request through to the SQS queue.
Integrate the API resource with our SQS Queue and create an integration mapping template to transform the incoming payload for our forwarding request to the SQS service.
Deploy and Test the API

Continue with me on my journey to complete these steps and end up with the following architecture, which uses zero-code.

To achieve a zero-code solution, we need to learn about the native AWS language of Velocity Templates and understand how data flows through the API Gateway service. As Ben and Farrah would say - writing code is easy, but being genuinely serverless takes work, and it's not always the work we want to do or is fun to do, but we need to challenge ourselves to leverage the cloud services and write less code!

How the API Gateway Data flow works

A quick high-level look at an API gateway request for a v1 REST API looks like this.

Each client request goes through the following stages for a simple un-authenticated request:

Method Request
Integration Request - where transformations occur for the integration request to SQS
Backend Service (Simple Queue Service in this case)
Integration Response - where transformations occur for the integration response from SQS
Method Response - Mapping of integration Response to Method response for the client.

Now that we know the steps for the API request through the gateway, we can start setting this up using the Serverless framework because that's my current go-to for deploying serverless solutions.

Create the SQS Queue

Creating the SQS queue is the easiest part of this solution. First, add the following to the resources section of the serverless YAML file. I prefer not to provide the QueueName when setting up SQS; this allows the serverless framework to name the resources using the usual naming standard of serviceName-stage-resourceName-randomString.

    sqsQueue:
      Type: AWS::SQS::Queue

Create the IAM Policy and Role

To create the IAM Role and Policy, we cannot use the standard IAM methods for the Serverless Framework since these focus on setting up IAM Roles and Policies for functions in your solution. This time there are no functions or Lambdas at all!
Add the role and policy details into the Serverless YAML file in the resources section.

Lines 23 - 34 enable API gateway to assume the role

Lines 35 - 44 define the actions allowed by the policy

Line 44 uses Cloudformation intrinsic function !GetAtt to get the Arn of the SQS Queue we create in the stack.

Create the Rest API

To create the Rest API, we need to create an AWS::Apigateway::Rest resource. Usually, the Serverless Framework would make this for you under the covers as it creates functions triggered by HTTP events. In the configuration for the RestApi, I have used ${self:custom.resourcePrefix} for the "Name", which is a variable I set up to use for the naming of Services throughout my Serverless YAML.

Create the API Resources and Integrate to our SQS

We create the API method using the "AWS::ApiGateway::Method" resource, which needs to look like this.

Creating the API method and integration is where all the work is in building out our zero-code solution and will break this down into the following steps for clarity:

Create the API Method and Responses (lines 60 - 70)
Create the integration - the easy bits (lines 71 - 79)
Create the Integration Request (lines 80 - 84)
Create the Integration Response (lines 85 - 99)

Create the API Method

Creating the Method resource is the easy bit. We set up the API Method on line 60 and Method responses on lines 64 - 69. The HTTP status codes returned by the Integration response must be configured in the Method responses so that the API gateway will not throw an error and return an "HTTP 500 internal server error".

On line 61, we use "!GetAtt apiGw.RootResourceId" to obtain the actual Id for the RestApi.

On line 62, we use "!Ref apiGw" to retrieve the RestApiId for our REST API.

We have to use these Cloudformation intrinsic functions here since we want the values after AWS has created them during the stack deployment.

Create the integration - the easy bits

I will break this down by the lines to make it easier to follow and explain each step.

Line 71 defines the HTTP Method API gateway will use when calling the Integration endpoint; in this case, we want to use POST since we are sending data to SQS.

Line 72 defines the type of integration; in this case, we are doing a direct service integration, so will use "AWS". Using AWS means using an internal integration that allows us to transform the payload rather than "AWS_PROXY", which will proxy the request directly to the integration service leaving the body untouched.

Line 73 defines the IAM Role the API Gateway will assume while performing the integration.

Line 74 defines the URI of the action we want to invoke. The Uri action we need to invoke will look like "arn:aws:API gateway:us-east-1:sqs:path/1234567890/the_queue_name", so we are using the Cloudformation intrinsic function "Join" to create this for us.

Once we have these attributes configured, we have defined the core details on the service we are integrating, the "How".

Create The Integration Request

Line 80 defines the PassthroughBehaviour - I like to use "NEVER" so that if we receive an unexpected request "Content-Type", then API gateway will respond with an HTTP 415 error - Unsupported Media Type, which I think is desirable.

Line 81 - 82 defines the request parameters we send to SQS; we will map our request into x-www-form-urlencoded values when sending our POST to SQS.

Lines 83 - 84 define how we will transform the incoming body and send it to SQS; this is a simple VTL template setting the Action=SendMessage and MessageBody=$input.JSON($), which is the JSON body of the request to be sent to the SQS queue.

Create the Integration Response

So far, we have created the API, formed the integration request and sent it to SQS; now, we have to deal with the SQS service response and map a response back to our caller using the "IntegrationResponses" attribute. The value expected here is a MAP of HTTP status codes and VTL template snippets to transform the HTTP response we get from SQS. In this demo, I have mapped the following response codes - 200, 404 to responses that will look something like the following:

HTTP 200 Response

We use a VTL template to transform the SQS response into a payload that we want to return rather than the entire integration payload.

{  
    "request_id": "ea87ccb1-92d3-51ff-b9f0-956ad19844d7",  
    "message_id": "173689a1-6b40-492e-9e6c-338c13b70f16", 
    "accepted": "ok" 
}

HTTP 404 Response

{
    "error_code": 404,
    "error_message": "Unable to complete Request"
}

Deploy and Test the API

After creating the SQS Queue and a new REST API, we need to deploy our API to a stage so our clients can call the API.
Lines 101 - 108 define the Stage Deployment of our API and will make the API available.
Important Note: This resource will only work for the first deployment; any change to your REST API will not deploy as part of a stack update. I searched far and wide for a solution to this behaviour and landed on the need to run a post-stack update API deployment using the console or AWS CLI.

$ aws apigateway create-deployment --region <region> \
    --rest-api-id <api-id> \
    --stage-name <stage-name>

The Wrap-Up

Writing a Zero-Code Serverless integration has been an evolving journey, and I have shared the details of where I ended up in this article. I have learned a lot about AWS along the way and encourage you to explore and try code-less integrations in your solutions as a way of levelling up your journey to serverless nirvana. What I have shared today is not production-ready - I am still exploring these patterns to make them more resilient and reliable for live use.

Check out the real working Serverless Solution (No code included) on my GitHub repository here