Forem: Anita Andonoska

Choosing the Right Configuration Management Solution in AWS

Anita Andonoska — Sun, 02 Feb 2025 21:13:01 +0000

In this blog post, we’ll walk through the best practices and strategies for managing configuration data in AWS.

Why Configuration Management is Important

Configuration management is an essential part of any modern infrastructure, especially when dealing with cloud services like AWS. Whether you are managing a simple application or a complex microservices architecture, handling configuration files correctly is crucial to maintaining system reliability, scalability, and security.

Configuration data refers to environment-specific configurations (such as API keys, database URLs, secrets, etc.) that applications require to function correctly, as well as application-specific settings related to the business logic within your system. Managing these files effectively ensures:

Consistency: Configuration across environments (e.g., development, staging, production) remains consistent.
Security: Sensitive data is protected from unauthorized access.
Flexibility: Enabling automated updates and scaling without manual intervention.
Without proper management, configuration files can become a source of errors, security risks, and inefficiencies.

Below we’ll cover different options for managing your configuration data in AWS.

AWS Systems Manager Parameter Store

AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management.

Security: You can store sensitive information, such as API keys or database credentials, and encrypt them.
Versioning: Track changes over time, making it easier to roll back when needed.
Access Control: Leverage IAM roles to define who can access or modify specific parameters.
Integration: Easily integrate with EC2 instances, Lambda functions, and ECS containers.

AWS Secrets Manager for Sensitive Data

For particularly sensitive data like database credentials, API tokens, or SSH keys, AWS Secrets Manager is the recommended service. It allows you to securely store and automatically rotate secrets.

Automatic Rotation: Secrets Manager can automatically rotate secrets for supported services or trigger custom rotations for other services.
Audit Logging: Every access to secrets is logged through AWS CloudTrail, providing an audit trail for security and compliance.
Fine-Grained Access Control: Use IAM policies to control access to specific secrets.

Custom-made solution with S3 or DynamoDb

When you require a more tailored solution for storing your configuration data, both Amazon S3 and DynamoDB are good options. S3 is well-suited for handling large data like JSON files, providing scalable storage with the ability to manage versions and control access through IAM policies. It’s ideal for storing files that don’t require frequent updates or complex queries. On the other hand, DynamoDB is better for configuration data that may need frequent updates or fast access, as it supports low-latency reads and writes.

Best practices on configuration management

Secure Sensitive Data: Sensitive configuration values, such as database passwords, must be encrypted and protected. Use AWS Secrets Manager and Systems Manager Parameter Store to encrypt values and enforce access control with IAM. Avoid storing sensitive data directly in code repositories or plaintext configuration files.
Use Environment-Specific Configurations: Separate configurations for different environments (development, staging, production) to avoid accidental misconfigurations. You can use different Parameter Store hierarchies or naming conventions to manage configurations for each environment.
Version Control Configuration Files: Track changes to configuration files by storing them in AWS S3 or Git repositories. Version control allows you to rollback to previous configurations in case of issues and keeps an audit trail of changes over time.
Ensure Secure Access: Always follow the principle of least privilege when granting access to configuration data. Use AWS Identity and Access Management (IAM) roles and policies to control who and what can access configuration data, and consider using AWS Secrets Manager for sensitive configurations like API keys or database credentials.
Automate Configuration Updates: Manual management of configuration data can be cumbersome. Automation of configuration updates ensures configurations are deployed consistently and can be updated in a controlled manner, reducing the risk of manual errors.

Effective Error Handling Strategies for AWS Lambda

Anita Andonoska — Sun, 19 Jan 2025 21:48:28 +0000

In this blog post we are covering various aspects of managing and mitigating errors in Lambda functions by explaining how to deal with errors, and ensure the reliability of serverless applications.

Types of Errors in AWS Lambda

There are two types of errors that can happen when invoking an AWS Lambda function: invocation and function errors.

An invocation error occurs when the invocation request is rejected before your function is triggered. This can happen due to issues with the request, the caller, or the account, such as invalid requests (e.g., incorrect JSON format, invalid parameters), insufficient permissions for the caller to invoke the function, or exceeding the maximum number of instances allowed within the AWS account.

A function error, on the other hand, happens when your function code or the runtime it uses generates an error. This can be caused by exceptions within the code, errors returned from the code, or runtime issues such as timeouts.

Error Handling Strategies in AWS Lambda

Error handling in AWS Lambda involves several key mechanisms to ensure smooth operation and issue resolution.

Function Code

One of the primary strategies is using try-catch blocks within Lambda code to catch exceptions and handle errors gracefully. Logging errors through Amazon CloudWatch Logs is also crucial, as it helps in identifying and diagnosing issues effectively. Additionally AWS X-Ray can be used to trace function executions and identify performance bottlenecks or errors in downstream services.

Retries

Errors can occasionally occur due to temporary issues, and retries can be helpful in such cases. For synchronous invocations, like APi Gateway there is no auto-retry mechanism. For asynchronous invocations, like S3 events, Lambda retries a failed asynchronous invocation up to two times. For event source mappings that read from streams, the entire batch of items is retried in case of an error. If errors persist, processing of the affected shard is blocked until the issue is resolved or the items expire. For event source mappings that read from a queue, you can control the retry interval and the destination for failed events by configuring the visibility timeout and redrive policy on the source queue.

Lambda Destinations

Lambda Destinations provide a method to send the results of function executions, whether successful or failed, to other AWS services like SQS, SNS, or EventBridge for further processing and monitoring. With Destinations, you can route asynchronous function results to a destination resource as an execution record without needing to write extra code. Compared to dead letter queues, destinations provide more useful capabilities by passing additional function execution information, including code exception stack traces, to more destination services.

Dead Letter Queues

As an alternative to an on-failure destination, you can configure your function with a dead-letter queue to retain discarded events for later processing. A dead-letter queue functions similarly to an on-failure destination, handling events that fail all processing attempts or expire without being processed. You can select either an Amazon SQS standard queue or an Amazon SNS standard topic for your dead-letter queue.

Best Practices for Error Handling in AWS Lambda

Best practices for error handling in AWS Lambda involve several key approaches to ensure robust and efficient function execution.

First, ensuring idempotency is crucial, as it allows Lambda functions to safely handle retries without unintended side effects. Additionally, graceful error handling is important; this includes properly managing known exceptions and providing clear, meaningful error messages to aid in troubleshooting. Optimizing function timeout settings is another best practice, as it helps balance performance and prevents unnecessary errors due to timeouts.

For specific use cases, implementing custom error handling logic can offer more control over how errors are managed and resolved. Finally, writing clean and detailed logs is essential for debugging, as comprehensive logs provide valuable insights into function execution and help identify and resolve issues quickly.

Data Streaming Hands-On: High-scale data streaming with AWS Kinesis Data Streams

Anita Andonoska — Thu, 21 Mar 2024 18:58:40 +0000

In my previous post, we saw how to create a data streaming app on AWS, with Kinesis Data Streams and Lambda. Here we’ll see high-scale data streaming in action by exploring Kinesis Data Streams on-demand scaling.

Kinesis Data Streams Scaling

A shard serves as a base throughput unit of Kinesis Data Streams. A shard supports 1 MB/s and 1,000 records/s for writes and 2 MB/s for reads. By default, new data streams created with the on-demand capacity mode have 4 MB/s of write and 8 MB/s of read throughput. That means we have 4 shards, each one allowing 1MB/second and 1000 records per second for writes, and 2MB/seconds for reads.

By default, On-Demand capacity mode can automatically scale up to 200MB/s of write and 400MB/s of read throughput. That is the default limit per account but you can request a limit increase through AWS Support to enable your On-Demand stream to scale up to 1 GB/s write and 2 GB/s read throughput.

According to the information provided in this AWS blog, scaling in on-demand capacity mode occurs at the individual shard level. When the average utilization of an ingest shard reaches 50% (0.5 MB/s or 500 records/s) within a minute, the shard is split into two. If random values are used as partition keys, all stream shards will have event traffic distribution, and scaling will occur simultaneously across all shards. Depending on the number of shards being scaled, it can take up to 15 minutes to split the shards.

As the load increases, Kinesis Data Streams adjusts the number of shards in the stream by monitoring the ingest throughput at the shard level. Nevertheless, under certain circumstances, producers might encounter “WriteThroughputExceeded” and “Rate Exceeded” errors, even when operating in on-demand capacity mode. According to AWS documentation, you may encounter "ProvisionedThroughputExceeded" exceptions if your traffic increases by more than double the previous peak within a 15-minute window. Further information on various scaling scenarios can be found here.

Load Generator app

I aimed to replicate the scenario of multiple producers sending data to the stream, and to achieve this, I developed a data streaming application for load generation. This application is built with Lambda functions serving as producers, utilizing Kinesis Data Streams with on-demand capacity, and Lambda functions as consumers. For a deeper understanding of the setup, you can refer to the specifics outlined in my earlier publication.

To generate the load, an additional Lambda function called the Producer Orchestrator has been introduced. This Orchestrator function is configured to accept three parameters: the duration of the load in milliseconds, the number of producers, and the number of records each producer will push to the Data Streams service. Depending on the specified number of producers, multiple instances of Lambda producers will be created. To ensure the data to be evenly distributed across the shards, a unique partition key is assigned to each record.

The code for the app can be found here.

Testing

As previously mentioned, scaling occurs when the average ingest shard utilization reaches 50% (equivalent to 0.5 MB/s or 500 records/s) within a minute. To initiate the testing, I began with a small load to align with the initial capacity of the stream, set at 4MB/s of write throughput. Initially, I started with 4 producers, each generating 1K records and then reached 80 producers with 6K records each. As the load increased, the number of shards scaled rapidly, reaching 12, 28, 60, 96, and ultimately 100 shards. This scaling was triggered by a brief surge in load lasting only a minute or two. My objective was to observe the scaling process, hence the load increase was not gradual, resulting in some records being throttled.

Summary

The testing conducted on scaling scenarios in Kinesis Data Streams revealed that scaling occurs swiftly when the average ingest shard utilization reaches the predefined threshold. The stream's capacity quickly adapted, with the number of shards increasing significantly in response to increased load. Even a brief (around 1 minute) surge in load triggered scaling actions, demonstrating the auto scaling capability of Kinesis Data Streams.

Streaming Services Review: AWS Kinesis Data Streams vs Azure Event Hubs

Anita Andonoska — Mon, 22 Jan 2024 18:05:32 +0000

In this post we’ll provide a comprehensive overview of AWS Kinesis Data Streams and Azure Event Hubs, covering various aspects of both services.

General overview

Data streaming (also known as event streaming) is a continuous flow of data generated by thousands of sources. Sources of data stream can be application logs, clickstream data from websites and mobile devices, telemetry data from IoT devices, real-time location tracking etc. (more info can be found in my previous post).

Azure offers Event Hubs as their streaming service: “Azure Event Hubs is a cloud native data streaming service that can stream millions of events per second, with low latency, from any source to any destination.” In Azure you first create Event Hubs namespace which is a management container for event hubs. In the namespace you create one or more event hubs. The data records within Event Hub are organized into partitions.

AWS offers Kinesis Data Streams: “Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams at any scale.” In Kinesis, you create the stream as a single resource. The data records within Kinesis data stream are organized into shards.

Capabilities

Azure Event Hubs is a multi-protocol event streaming engine that natively supports AMQP, Apache Kafka, and HTTPs protocols. Event Hubs is compatible with Apache Kafka, and it enables you to run existing Kafka workloads without any code changes. Event Hub integrates with Azure Functions, Azure Spring Apps, Kafka Connectors, and other data analytics platforms and technologies such as Databricks, Apache Spark and Apache Flink.

AWS Kinesis Data Streams is exposed through HTTPs protocol. AWS Kinesis Data Streams integrates with other AWS services and third party applications for sending and consuming data. On the producer side Kinesis Data streams integrates with Amazon Aurora, DynamoDb, CloudWatch, EventBridge, etc., and with third party services, like the integration with Kafka. To consume the data, there are direct integrations to Kinesis Data Analytics, AWS Lambda, EventBridge, SQS, SNS, Glue, etc. and there are third party integrations to Databricks, Apache Spark, etc.

Both services allow event size up to 1MB.

Pricing Models

As usual, Microsoft offers tiered pricing with Basic, Standard, Premium and Dedicated pricing models for Event Hubs as well. The higher cost models offer more throughput capacity and there are some differences in features available. Throughput units are pre purchased and are billed per hour. You are charged for throughput units and number of ingress events.

Kinesis Data Streams has two capacity modes: on-demand mode, where AWS manages the capacity and provisioned mode, where you specify the number of shards for your application.
In on-demand mode, pricing is based on the volume of data ingested and retrieved along with a per-hour charge for each data stream in your account. In provisioned mode you’re charged for each shard at an hourly rate and also pay for records written into your Kinesis data stream.

Scaling and Performance

Scaling in Event Hub depends on throughput units (standard tier) or processing units (premium tier) and partitions. A single throughput unit allows for up to 1 MB per second or 1,000 events per second ingress, and up to 2 MB per second or 4,096 events per second egress. Beyond the capacity of the purchased throughput units, ingress is throttled. A standard Event Hubs namespace has a limit of 40 throughput units, and these are shared across all event hubs in that namespace.

The auto-scaling or as it is called in Azure, auto-inflate feature of Event Hubs, automatically scales up by increasing the number of throughput units, to meet usage needs. You need to initially set the desired throughput units. The Auto-inflate feature of Event Hubs automatically scales up by increasing the number of TUs, to meet usage needs. Increasing TUs prevents throttling scenarios where data ingress or data egress rates exceed the rates allowed by the TUs assigned to the namespace. Auto-inflate doesn’t automatically scale down the number of TUs when ingress or egress rates drop below the limits.

Kinesis Data Streams by default, if you choose on-demand mode, scales capacity automatically, freeing you from provisioning and managing capacity. Kinesis Data Streams monitors your data traffic and scales the shard count up or down depending on traffic increase or decrease. You can choose provisioned mode if you want to provision and manage throughput on your own.

By default, new data streams created with the on-demand capacity mode have 4 MB/s of write and 8 MB/s of read throughput. As the traffic increases, data streams with the on-demand capacity mode scale up to 200 MB/s of write and 400 MB/s read throughput. You can request a quota increase up to 2 GB/s write and 4 GB/s read capacity.

Data Consumption

With Event Hubs the consumption of data is done through consumer groups. Consumer group is a logical grouping of consumers that read data from an event hub. It enables multiple consuming applications to read the same streaming data in an event hub independently at their own pace with their offsets.

In Kinesis you can choose between shared fan-out and enhanced fan-out consumer types to read data from a Kinesis data stream. When a consumer uses enhanced fan-out, it gets its own 2 MB/sec allotment of read throughput, allowing multiple consumers to read data from the same stream in parallel, without contending for read throughput with other consumers. You should use enhanced fan-out if you have, or expect to have, multiple consumers retrieving data from a stream in parallel. There are additional charges for using the Enhanced Fan-Out feature.

Data Retention and Durability

Published events are removed from an event hub based on a configurable, timed-based retention policy. The default value and shortest possible retention period is 1 hour. For Event Hubs Standard, the maximum retention period is 7 days. For Event Hubs Premium and Dedicated, the maximum retention period is 90 days. If you need to archive events beyond the allowed retention period, you can have them automatically stored in Azure Storage or Azure Data Lake by turning on the Event Hubs Capture feature.

Kinesis Data Streams have a default retention period of 24 hours that can be extended to 7 days. There is also long-term data retention greater than seven days and up to 365 days that lets you reprocess old data. You can also deliver data stored in Kinesis Data Streams to Amazon S3, Amazon OpenSearch Service, Amazon Redshift, and custom HTTP endpoints using its prebuilt integration with Kinesis Data Firehose.

Conclusion

Both services in general offer similar features, like auto-scaling, data retention and durability etc. Event Hubs support more protocols like AMQP and Apache Kafka.

There is a difference in scaling and performance. AWS has a simple model for this, you either choose on-demand and the service scales up or down based on traffic, or you handle the capacity yourself with provisioned mode. With Event Hubs even though there is auto scaling, it does not scale down, so you have to manage this yourself.

Data Streaming Hands-On: Building Kinesis Data Streams App

Anita Andonoska — Mon, 01 Jan 2024 19:17:38 +0000

This is a step-by-step guide that will walk you through the process of building an AWS Kinesis Data Streams application.

Set Up AWS Kinesis Data Streams service

When creating AWS Kinesis Data Streams service there are two capacity modes to choose from:

On demand, should be used when your data stream’s throughput requirements are unpredictable and variable. With on-demand mode, your data stream’s capacity scales automatically.
Provisioned: should be used when you can reliably estimate throughput requirements of your data stream. With provisioned mode, your data stream’s capacity is fixed.

For this example I am using a Kinesis Data Streams service with On-Demand capacity mode, so the data stream’s capacity will scale automatically. At time of creation I went with default settings for the Kinesis Data Stream service.

Build a Data Producer

We need to create an application that will simulate data production. To send the data we’ll utilize AWS Kinesis SDK so that we can interact with the Kinesis API.

In my example I am using AWS SDK for .NET, and I have created a Lambda function as a producer, but this can be any type of application or script, and of course it can run outside of AWS.

The AWS Kinesis SDK offers two methods for sending the data, PutRecord for sending a single record, and PutRecords for sending a batch of records. For the Lambda function to be able to send data to Kinesis Data Streams, it is needed to have the PutRecord and/or PutRecords permission assigned based on which API method you’ll be using. Here we’ll show the usage of the PutRecords method since sending a batch is more fun.

Each PutRecords request can support up to 500 records. Each record in the request can be as large as 1 MiB, up to a limit of 5 MiB for the entire request, including partition keys. To send a batch of records we’ll need to make sure that a single PutRecords request does not go over the maximum number of records or the maximum size limit, see the code sample below.

Build a Data Consumer

The Amazon Kinesis Data Streams integrates with many AWS and third party services for consuming the data (more info in my previous post), also a custom application can be created to read the data by utilizing Kinesis Data Stream API.

Here as a data consumer, I have created an AWS Lambda that integrates with the Kinesis Data Stream. The Kinesis Data Stream service is added as a trigger for the function. Also proper permissions (kinesis GetRecords, ListStreams, DescribeStream etc.) are needed to be assigned to the function so that it can be triggered by Kinesis Data Stream service .

In the code below the messages are being read from the stream.

Monitor and Troubleshoot

For monitoring and troubleshooting the Data Streams Service offers an automatic dashboard:

The dashboard is based on metrics that automatically are sent to CloudWatch from the Data Streams service. Additionally you can leverage the metrics to create your own dashboards or create alarms.

Beside this I am using CloudWatch logs for the custom logging from the producer and consumer Lambda functions.

Conclusion

In the preceding discussion, we’ve walked through the steps of creating a Data Streaming app, tackling the generation and transmission of streaming data, and finally, the consumption of that data.

You can find the complete code on my GitHub repository.

FaaS Comparison: AWS Lambda vs Azure Functions

Anita Andonoska — Mon, 16 Oct 2023 19:38:01 +0000

In this post we’ll explore the different features and capabilities of AWS Lambda and Azure Functions.

Intro

FaaS (Function as a Service) is a cloud computing service model where cloud providers allow users to run individual functions or pieces of code in response to specific events. These events may include changes in state or an update, such as a user placing an item in a shopping cart on an ecommerce website. These functions, also known as serverless functions or serverless compute, are small units of compute that are designed to perform a specific task. They are event-driven and executed in stateless environments. When a specific event occurs (such as an HTTP request, database change, or file upload), the function is automatically invoked, and the cloud provider manages all the necessary infrastructure, including server provisioning, scaling, and maintenance.

The serverless functions were first introduced in November 2014 when Amazon Web Services introduced AWS Lambda. Since then, other cloud providers like Microsoft Azure (with Azure Functions) and Google Cloud Platform (with Google Cloud Functions) offer similar services.

Setup

To create a new AWS Lambda function is very simple and intuitive, you just need to select the Runtime (programming language you want to write the function code) and architecture (x86_64 or arm64). Additionally it is important to choose a memory size for the function. The default one is 128 MB and the max is 10GB. The memory size determines the computing power of the function. Lambda allocates CPU power in proportion to the amount of memory configured. And that’s all you need to do to create your first function!

To create a new Azure Function, first you need to create a Function App. A Function App can hold many Azure Functions, letting you group functions as a logical unit for easier management, deployment, scaling, and sharing of resources. At the time of creation of the Function App, beside specifying the runtime, you need to specify Operating System Linux/Windows, choose the Hosting Plan and choose a storage account. The chosen Hosting Plan is created as an additional resource in your Azure infrastructure. Multiple Function Apps can share the same plan.

Azure offers the following hosting plans

Consumption, this is the only fully serverless hosting plan, where instances of the Azure Functions host are dynamically added and removed based on the number of incoming events. The max memory per instance is 1.5GB.
Functions Premium, offers scaling and network isolation, ideal for workloads running continuously. With the Premium plan there is at least one always active instance. The max memory per instance is 3.5–14 GB.
Dedicated, fully isolated and dedicated environment. These dedicated compute resources are analogous to the server farm in conventional hosting. The max memory per instance is 1.75–14 GB.

Function organization

In AWS Lambda, you define and deploy individual functions. Each function is a separate entity, and you manage them individually. Each function operates independently, and they do not share a common state unless you integrate external services for that purpose. AWS Lambda provides granular control over individual functions, allowing you to set specific memory allocations, timeouts, and environment variables for each function.

In Azure Functions, functions are organized into Function Apps. A Function App is a logical container for functions. Functions within a Function App can share common resources, such as application settings and connection strings. Functions within a Function App share the same resources allocated to the Function App, which includes memory and CPU.

Triggers (and bindings)

A trigger is a resource you configure to allow another service to invoke your function when certain events or conditions occur. Common use cases are: invoke the function on new upload in object storage, HTTP request, incoming messages in a queue, run function on schedule etc. Triggers are common for both AWS Lambda and Azure Functions.

What is different at Azure Functions is bindings. Binding to a function is a way of declaratively connecting another resource to the function; bindings may be connected as input bindings, output bindings, or both. Data from bindings is provided to the function as parameters. Triggers and bindings let you avoid hardcoding access to other services. Your function receives data (for example, the content of a queue message) in function parameters. You send data (for example, to create a queue message) by using the return value of the function.

Scaling

The Lambda service runs your function only when needed and scales automatically. You only pay for the compute time that you consume — there is no charge when your code is not running. When your function is invoked more quickly than a single instance of your function can process events, Lambda scales up by running additional instances. When traffic subsides, inactive instances are frozen or stopped. You pay only for the time that your function is initializing or processing events.

There is a default quota of 1000 concurrent Lambda instances on Account level, that can be extended to tens of thousands. If there is a sudden short period of high activity AWS Lambda allows temporary bursts of concurrent executions above your specified concurrency limit. AWS Lambda automatically scales function concurrency in response to the incoming load. Burst concurrency is a feature that enables your functions to handle temporary increases in traffic efficiently without the need to manually adjust your concurrency limits.

For Azure Functions, the scaling depends on the hosting plan:

Consumption, scales out automatically, to 200 instances for Windows and 100 for Linux
Premium, both manual and autoscaling, max to 100 instances Windows/Linux within a single plan (differs per region)
Dedicated, both manual and autoscaling. Using an App Service plan, you can manually scale out by adding more VM instances. You can also enable autoscale.

Duration

Duration refers to the amount of time that your function code spends processing an event, and it does not include the cold start time.

AWS Lambda functions can be configured to run up to 15 minutes per execution. You can set the timeout to any value between 1 second and 15 minutes.

For Azure Functions, the duration also depends on the hosting plan:

Consumption, the default is 5 minutes, and maximum is 10 minutes
For Premium and Dedicated, the default duration is 30 minutes, and the maximum duration is unlimited (interruptions may occur, up to 60 minutes guaranteed)

Logs and monitoring

When using Lambda, you are responsible only for your code. Lambda manages the compute fleet that offers a balance of memory, CPU, network, and other resources to run your code.

Lambda performs operational and administrative activities, including managing capacity, monitoring, and logging your Lambda functions. Lambda automatically monitors Lambda functions on your behalf and reports metrics through Amazon CloudWatch.

For Azure Functions, logs and monitoring dashboards are not available out of the box. You need to enable App Insights when you create the Function App or enable it afterwards. With this you add additional resources, App Insights, to your infrastructure, which comes with additional costs as well. Also you will have to get familiar with Kusto, the query language for the App Insights Logs.

Pricing

With AWS Lambda you only pay for what you use. Charges are based on the number of requests and the duration of code execution. Duration is calculated from the time your code begins executing until it returns or otherwise terminates. The price depends on the amount of memory you allocate to your function.

For Azure functions, charges are based on the number of executions and execution time, with different pricing tiers based on the plan chosen:

Consumption Plan, Billing is based on number of executions, execution time, and memory used. Usage is aggregated across all functions within a function app.
Premium Plan, Billing for the Premium plan is based on the number of core seconds and memory allocated across instances. This billing results in a minimum monthly cost per active plan, regardless if the function is active or idle.
Dedicated Plan, You are billed only for the plan, regardless of how many function apps or web apps run in the plan.

Summary

The above outlines the major features and capabilities of both services. The services have some similarities in features common for serverless functions, but on the other hand they are designed in very different manners.

AWS Lambda is a pure serverless service and setting it up is really straightforward, the user just needs to select the proper memory size.

For Azure Functions the available capabilities depend on the hosting plan you choose. If you are looking for a serverless option then it is the Consumption plan, unless you hit some limitations and have to go for the non-serverless plans.

Demystifying Amazon Kinesis Data Streams Concepts

Anita Andonoska — Wed, 30 Aug 2023 19:36:46 +0000

In the previous post we’ve explored what data streaming is, and here we’ll explore Amazon Kinesis Data Streams service.

What is Amazon Kinesis Data Streams

Amazon Kinesis Data Streams is a fully managed streaming data service that collects and stores data. It is a highly scalable and durable data ingestion pipeline. It can continuously collect and aggregate data from a wide range of sources, including IoT devices, applications, servers, and log files. It can continuously capture and store terabytes of data per hour from hundreds of thousands of sources. Data is stored in the order it was received and can be replayed indefinitely within the retention period.

It enables real-time data processing, allowing users to analyze and react to incoming data streams as they happen. This is crucial for applications that require low-latency data processing and quick decision-making.

Key concepts

Stream
A stream is the fundamental concept in Amazon Kinesis Data Streams, it is an ordered sequence of data records.

Shard
A shard is a sequence of data records in a stream. Shards are used to scale the stream and distribute data across multiple instances for parallel processing.

Data records
Data records are the individual units of data in a stream. Each data record consists of a data payload and a sequence number, which uniquely identifies the record within its shard.

Producers and Consumers
An application that sends records to a Kinesis Data Streams is called a producer. Applications and services that read and process data from Kinesis Data Streams are called consumers.

Capacity mode
Amazon Kinesis Data Streams offers two capacity modes for data ingestion:

On-Demand mode. Kinesis Data Streams scales capacity automatically by managing the shards to provide the necessary throughput. You pay only for the actual throughput used, and Kinesis Data Streams automatically scales up or down to accommodate your workload throughput.
Provisioned mode. You can choose provisioned mode if you want to provision and manage throughput on your own. In provisioned mode, you specify the number of shards for the data stream. The total capacity of a data stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a data stream as needed, and you pay for the number of shards at an hourly rate.

Sending and Consuming Data

AWS Kinesis Data Streams integrates with other AWS services and third party applications for sending and consuming data.

To mention a few, on the producer side Kinesis Data streams integrates with Amazon Aurora, DynamoDb, CloudWatch, EventBridge, etc. From the third party services, the most interesting one (for me) is the integration with Kafka.

To consume the data, there are direct integrations to Kinesis Data Analytics, AWS Lambda, EventBridge, SQS, SNS, Glue, etc. Through Kinesis Firehose the data can be delivered to different destinations like S3, DynamoDb, OpenSearch, etc. There are third party integrations to Databricks, Apache Spark, etc.

You can also build custom producer and consumer applications by using Amazon Kinesis APIs.

Use cases

Kinesis Data Streams enables rapid and continuous data flow from producers and then continuous processing of the data, either by transforming it before emitting to a data store, running real-time metrics and analytics, or deriving more complex data streams for further processing.

Common use cases for Amazon Kinesis Data Streams include real-time analytics, monitoring and alerting, clickstream analysis, fraud detection, IoT data processing, and log data aggregation.

Data Streaming on AWS

Anita Andonoska — Sat, 17 Jun 2023 10:56:49 +0000

In this post we’ll explore what is data streaming and take a look at AWS data streaming services.

Data Streaming

With data streaming, high volumes of high velocity data from different sources can be ingested, processed and analyzed in real-time. The data streaming service captures the data sent from the different sources, stores it for a set duration of time, and the data can be replayed indefinitely for this duration period. Multiple different applications can simultaneously and independently consume the data.

There are different approaches to process streaming data: batch processing and stream processing. With traditional systems the data is first stored and then processed. With stream processing services the data can be consumed, enriched and analyzed in motion.

Batch Processing vs Stream Processing

Batch processing system, takes a large set of data as input, runs a job to process it, and it produces output data. The input is bounded, it has known and finite size, so that the batch process knows when it has finished reading the input. But in reality most of the data is unbounded as it arrives gradually over time, so with the batch process the data is divided in chunks of fixed duration, e.g. per hour or per day. Usually batch jobs are scheduled to run periodically, and often it takes some time for the job to complete, from minutes to several days. Changes in the input are only reflected in the output, hours or days later when the job completes. Batch processing is suitable for historical data analysis, machine learning etc.

With stream processing the data is processed continuously in real-time, even before it is stored. Like batch processing, the stream processing consumes input data and produces output, but it does this on events shortly as they happen. The data can be processed on a record-by-records basis or over sliding time windows. Stream processing is intended for time sensitive applications, where data needs to be processed as soon as it arrives. Requires latency in the order of seconds or milliseconds. It is suitable for real-time stock trading applications, social media feeds, alerting systems etc.

Both approaches can even be combined, by having a real-time layer and a batch layer. Data is first processed to extract real-time insights and then persisted to storage from where it can be used for different batch processing scenarios.

AWS Streaming Services

Data streaming on AWS is enabled by the Amazon Kinesis family of services.

Amazon Kinesis Data Stream is a fully managed streaming data service that collects and stores data streams. It can continuously capture and store terabytes of data per hour from hundreds of thousands of sources. Data is stored in the order it was received and can be replayed indefinitely within this period. You can build stream processing applications that consume the data from Kinesis streams with Kinesis Data Analytics, Lambda or Spark Streaming.

Amazon Kinesis Data Firehose loads the data streams into AWS data stores like S3, Redshift, ElasticSearch etc. It loads the streaming data into storage destinations with zero administration.

Amazon Kinesis Data Analytics enables real-time analytics on the data stream. It enables to process data continuously and get insights in seconds or minutes.

More details on batch and stream processing can be found in Designing Data-Intensive Applications by Martin Kleppmann. Data streaming scenarios and examples can be found in this AWS Whitepaper.

Event-Driven Architectures on AWS

Anita Andonoska — Thu, 23 Feb 2023 19:11:52 +0000

In this post we’ll go through the main components of an event-driven architecture and explore AWS services that fit in event-driven architectures.

What is it?

Event-driven architecture (EDA) represents a system of loosely coupled components that exchange information between each other asynchronously through events. An event represents a change in state (e.g. registration form is submitted, booking has been made).

In traditional request-response architectures, the sender sends a request and waits for the response prior moving to the next task. Additionally, both the sender and the receiver are directly impacted by any failures that occur on any side, e.g. if there is a failure at the receiver the sender has no option of how to communicate and send the request.

In event-driven architecture, the component that generates an event can immediately proceed with its next task. Then different components of the system can react to the emitted event.

How does it work?

Event-driven architecture is based on asynchronous communication between system components via events. The event is immutable, meaning the event cannot be changed after its creation. Each time there is a change in an entity’s state, a new event is generated. With the immutable events in a distributed system there are no changes across components to keep in sync. Events are observed, not directed. A component that emits an event has no particular destination nor awareness of downstream components that may consume the event.

Event-driven architecture consists of event producers, event routers and event consumers:

Event producers publish events to a router.
Event brokers mediate between producers and consumers.
Event consumers are downstream components that react to events.

The event broker receives events, stores them in a queue or event stream and provides them for consumption to other system components.

Producers and consumers are decoupled, which allows them to be scaled, updated and deployed independently. The producer is not aware which consumers, if any, are listening for the events. The consumers can process the event on their own schedule. The consumers are also decoupled from each other and every consumer sees all the events. This is in contrast to message-driven architecture where a message is intended for one consumer and deleted after it is being consumed.

Event-driven architectures on AWS

At the core of every event-driven architecture is the event broker, which can be an event router or an event stream. For event routers there are two main types: event buses and event topics. In AWS, these components are represented by Amazon Event Bridge as event bus and Amazon Simple Notification Service (SNS) as event topic.

EventBridge is an event bus service. It applies rules to route events from event sources to different targets. Targets can include AWS services such as AWS Lambda, Step Functions, and Amazon Kinesis, or any HTTP endpoint through EventBridge API destinations.

SNS is a service that enables publishing messages from an application and immediately delivers them to subscribers or other applications. Publishers communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. Clients can subscribe to the SNS topic and receive published messages.

For event streaming, Amazon Kinesis Data Streams can be used as an event-streaming service. It is a highly scalable service that allows multiple consumers reading from the stream and provides batch processing instead of an individual event at a time.

Here you can find more details on event-driven architectures, event-driven architectures in serverless applications, building event-driven microservices and benefits of migrating to event-driven architecture.

Why choose Amazon Web Services: technical point of view

Anita Andonoska — Thu, 19 Jan 2023 20:18:09 +0000

In this post, based on my experience with Amazon Web Services I am sharing the reasons why to choose it as a cloud service provider.

Cloud computing has been in huge demand in the last few years. The choice you’ll make about the cloud service provider will determine the course of how your project evolves. This implies how easy it is to work with the services, whether the services scale as expected, etc.

Here are my top reasons why to choose AWS.

Simplicity

AWS is designed to allow developers to get comfortable with the platform quickly. You can focus on developing your applications, while AWS takes care of the infrastructure. This is because the services are designed around the idea of simplicity and consistency.

The AWS cloud currently offers 200+ services, for a wide range of technologies, industries, and use cases. But every service has a particular purpose and you would not end up in a confusing situation about which service to choose.

Also, there is consistency in how things are being done. For example, IAM is a service that is used to set access to services and resources across all of AWS. So when you need to setup access to a service or a resource you can do this in a consistent way by using IAM.

Out-of-the-box features

With every production-ready system, it is expected to have a logging mechanism. With many of the AWS services you just have to enable the logging option for the service and you get the logs in AWS CloudWatch, There is no need to provision or setup this logging service.

For example, with AWS Lambda you just need to give permission to the Lambda function to write logs to CloudWatch and that’s it.

It scales!

Most of the AWS services automatically handle scaling without any setup. That is the case for AWS Lambda. As the function receives more requests, automatically handles scaling the number of execution environments (until the account’s concurrency limit is met). You do not have to provision or setup anything.

Constantly improving

Over the years there have been released many improvements to the AWS services that lead to lower costs and better performance. Releasing new features is expected, but what is impressive is AWS devotion to further improve already existing features.

One such example is the release of a new S3 Intelligent-Tiering Archive Instant Access tier. This tier was automatically introduced to already existing S3 workloads that relied on S3 Intelligent-Tiering. With this new class, AWS automatically saved up to 68% on storage costs to customers. How cool is that?

Understanding Amazon S3 Storage Classes

Anita Andonoska — Wed, 23 Nov 2022 19:17:37 +0000

Amazon S3 offers a range of storage classes based on the data access patterns, resiliency, and cost requirements of your workloads. Understanding the S3 Storage classes is essential to optimally manage your costs and access to the S3 objects.

Intro

Each object in Amazon S3 has a storage class associated with it. S3 Storage Classes can be configured at the object level and a single bucket can contain objects stored across S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA. You can set a storage class on an S3 object when you create the object. Also, you can change the storage class of an existing S3 object. This can be done by copying the object with the new storage class. You can also use S3 Lifecycle policies to transition objects between storage classes automatically.

You pay for storing objects in your S3 buckets. The rate you’re charged depends on your objects’ size, how long you stored the objects during the month and the object’s storage class. You can estimate your costs using the AWS Pricing Calculator.

The storage classes can be split into groups based on the access pattern: frequent access, unknown or changing access, infrequent access, and archive access.

Frequent Access

S3 Standard is a general-purpose storage class for any type of data, typically used for frequently accessed data. If a storage class is not specified when the object is uploaded, S3 Standard class is the default class assigned to an S3 object. S3 standard has low latency and high throughput which makes it suitable for a wide range of use cases. Use the S3 Standard class when an object is accessed more than once a month and millisecond access is needed.

Unknown or changing access

For data with changing, unknown, or unpredictable access patterns, such as data lakes, analytics, or new applications, use S3 Intelligent-Tiering. S3 Intelligent-Tiering automatically moves your data between three low-latency access tiers optimized for frequent, infrequent, and rare access. S3 Intelligent-Tiering monitors access patterns and automatically moves objects that have not been accessed to lower-cost access tiers.

S3 Intelligent-Tiering automatically stores objects in three access tiers: a Frequent Access tier, an Infrequent Access tier, and a Archive Instant Access tier. Objects that are uploaded or transitioned to S3 Intelligent-Tiering are automatically stored in the Frequent Access tier. S3 Intelligent-Tiering works by monitoring access patterns and then moving the objects that have not been accessed in 30 consecutive days to the Infrequent Access tier. Any existing objects that have not been accessed for 90 consecutive days will automatically move to the Archive Instant Access tier. As data moves into a more cost-efficient storage class, you pay less for the storage and at the same time, you can still access the data immediately. Frequent, Infrequent, and Archive Instant Access tiers have the same low-latency and high-throughput performance as S3 Standard.

Additionally, for data that does not require immediate access, you can activate one or both of the archive access tiers: Archive Access and Deep Archive Access tier. To access the archive data you will need to restore the data.

You pay a monthly monitoring and automation charge per object stored in the S3 Intelligent-Tiering storage class to monitor access patterns and move objects between access tiers. There are no retrieval charges, and no additional tiering charges apply when objects are moved between access tiers.

Objects smaller than 128KB are not eligible for auto-tiering. They can be stored in S3 Intelligent-Tiering but will always be charged at the Frequent Access tier rates, and you are not charged for the monitoring and automation of these objects.

Infrequent access

Use S3 Standard-IA and S3 One Zone-IA storage classes for long-lived and infrequently accessed data, that require millisecond access. Amazon S3 charges a retrieval fee for these objects, so they are most suitable for infrequently accessed data.

S3 Standard-IA class stores the object data redundantly across multiple geographically separated Availability Zones. Use it for your primary or only copy of data that can’t be re-created. S3 Standard-IA offers the high durability, high throughput, and low latency of S3 Standard, with a low per GB storage price and per GB retrieval charge.

Unlike other S3 Storage Classes which store data in a minimum of three Availability Zones (AZs), S3 One Zone-IA stores data in a single AZ and costs 20% less than S3 Standard-IA. But, the data is not resilient to the physical loss of the Availability Zone. It’s a good choice for easily re-creatable data. You can also use it as cost-effective storage for data that is replicated from another AWS Region using S3 Cross-Region Replication.

Archive access

The Amazon S3 Glacier storage classes are purpose-built for data archiving and you can choose from three archive storage classes optimized for different access patterns and storage duration.

S3 Glacier Instant Retrieval, is built for long-lived archive data accessed once a quarter with instant retrieval in milliseconds. Compared to S3 Standard-IA, this class offers storage cost savings but higher data access costs. With S3 Glacier Instant Retrieval, you can save up to 68% on storage costs compared to using the S3 Standard-Infrequent Access (S3 Standard-IA) storage class, when your data is accessed once per quarter. S3 Glacier Instant Retrieval delivers the fastest access to archive storage, with the same throughput and milliseconds access as the S3 Standard and S3 Standard-IA storage classes.

S3 Glacier Flexible Retrieval, is built for archiving data that does not require immediate access and offers retrieval option from 1 minute to 12 hours. S3 Glacier Flexible Retrieval delivers low-cost storage, up to 10% lower cost (than S3 Glacier Instant Retrieval), for archive data that is accessed 1–2 times per year and is retrieved asynchronously. S3 Glacier Flexible Retrieval delivers the most flexible retrieval options that balance cost with access times ranging from minutes to hours and with free bulk retrievals.

S3 Glacier Deep Archive is Amazon S3’s lowest-cost storage class and supports long-term retention of data that may be accessed once or twice a year. It is designed for customers that retain data sets for 7–10 years or longer to meet regulatory compliance requirements. S3 Glacier Deep Archive can also be used for backup and disaster recovery use cases.

S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are not available for real-time access. You must first restore S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects before you can access them.

All the different ways to invoke an AWS Lambda

Anita Andonoska — Fri, 11 Nov 2022 21:15:00 +0000

In this post, we’ll go through the different ways to invoke an AWS Lambda function.
AWS Lambda can be invoked synchronously, asynchronously, or through event source mapping. Let’s explore each of them.

Synchronous invocation

With synchronous invocation, the caller waits for the function to process the event and return a response. Details about the function response, including errors, are included in the response body and headers. When invoked synchronously, Lambda sends the request directly to the function and then the function’s response is sent back to the caller. When you invoke a function directly, you are responsible to examine the response, determining whether an error has happened and whether to retry.

You can invoke Lambda functions directly using the Lambda console, a function URL HTTP(S) endpoint, the Lambda API, an AWS SDK, the AWS Command Line Interface (AWS CLI), and AWS toolkits. You can also configure other AWS services to invoke your function synchronously, like API Gateway, CloudFront (Lambda@Edge), Elastic Load Balancer, etc.

Asynchronous invocation

When you invoke a function asynchronously, you don’t wait for a response from the function code. You hand off the event to Lambda and Lambda handles the rest. Lambda handles retries and can send invocation records to a destination.

With asynchronous invocation, Lambda queues the event for processing and returns a response immediately. Lambda places the event in a queue and returns a successful response without additional information. A separate process reads events from the queue and sends them to your function. If an error occurs, AWS will automatically retry the invoke.

Several services invoke Lambda function asynchronously: Simple Storage Service (S3), Simple Notification Service (SNS), Simple Email Service, Event Bridge, CloudWatch Logs, etc.

Event source mapping

Lambda uses event source mapping to process items from a stream or queue. This invocation type is also known as poll-based invocation. You can use event source mappings to process items from a stream or queue in services that don’t invoke Lambda functions directly.

An event source mapping is a Lambda resource that reads events from an event source and sends them to your function in batches. Each event that your function processes can contain hundreds or thousands of items. You can configure the batching behavior by setting a batching window and batch size. A batching window is the maximum amount of time to gather resources into a single batch. Batch size is the maximum number of events in a single batch.

Lambda event source mappings process events at least once due to the distributed nature of its pollers. As a result, in rare situations, your Lambda function may receive duplicate events.

Lambda provides event source mappings for the following services: DynamoDb, Simple Queue Service (SQS), Kinesis, etc.

Conclusion

Depending on who invokes your function and how it’s invoked, scaling behavior and the types of errors that occur can vary.