Forem: Simranjeet Singh

Understanding Attribute Types in DynamoDB

Simranjeet Singh — Sat, 22 Jul 2023 06:14:01 +0000

In Amazon DynamoDB, attribute types define the nature and format of the data stored within attributes. DynamoDB provides a variety of attribute types to accommodate different data requirements and use cases. By selecting the appropriate attribute types, you can ensure efficient storage, retrieval, and querying of data. In this section, we’ll explore the attribute types supported by DynamoDB and discuss their characteristics.

Scalar Types:

String: Represents a sequence of Unicode characters with a maximum length of 400KB. Commonly used for storing textual data such as names, descriptions, or identifiers.
Number: Represents a numeric value, which can be either an integer or a floating-point number. Numbers can be positive, negative, or zero.
Binary: Represents binary data, such as images, audio files, or serialized objects. Binary attributes store raw byte arrays.

Set Types:

String Set: Represents an unordered collection of unique string values. Suitable for scenarios where you need to store multiple, distinct string values.
Number Set: Represents an unordered collection of unique numeric values. Useful for scenarios that require storing multiple, distinct numeric values.
Binary Set: Represents an unordered collection of unique binary values. Ideal for scenarios involving multiple, distinct binary data.

Document Types:

List: Represents an ordered collection of elements. Each element within the list can be of any DynamoDB data type, including other lists and maps. Lists are useful for representing ordered data structures.
Map: Represents an unordered collection of key-value pairs. The keys within a map must be unique, and the values can be of any DynamoDB data type, including other maps and lists. Maps enable flexible and nested data structures.

Each attribute within an item can have a specific data type. DynamoDB is schema-less, allowing different items within the same table to have different attributes and attribute types. This flexibility enables you to adapt to changing data requirements without modifying the table structure.

Choosing the appropriate attribute types is important for optimizing storage, query performance, and cost efficiency. By accurately representing the data type, you can leverage DynamoDB’s indexing and querying capabilities effectively.

Additionally, DynamoDB supports a rich set of operations and functions for manipulating attribute values, such as comparisons, filtering, concatenation, and more. These operations provide flexibility in working with attribute values and enable powerful query capabilities.

Conclusion:

Understanding attribute types in DynamoDB is crucial for accurately representing and manipulating data within your tables. By selecting the appropriate attribute types for your data, you can optimize storage, query performance, and cost efficiency. DynamoDB’s support for scalar types, set types, and document types offers a wide range of options to handle different data scenarios.

In the next article, we will explore how to model relationships between items in DynamoDB and discuss various strategies for handling one-to-one, one-to-many, and many-to-many relationships. Stay tuned for more insights and best practices on working with Amazon DynamoDB!

Originally published at https://awsmag.com on July 22, 2023.

DynamoDB Data Model: Tables, Items, and Attributes Explained

Simranjeet Singh — Sun, 16 Jul 2023 13:11:31 +0000

When working with Amazon DynamoDB, it’s essential to understand the data model it employs. DynamoDB, a fully managed No-SQL database service offered by Amazon Web Services (AWS), uses a schema-less and flexible data model, allowing for rapid and scalable application development. In this section, we’ll provide an introduction to DynamoDB’s data model, exploring the key concepts and components that form its foundation.

At the core of DynamoDB’s data model are three key elements: tables, items, and attributes. Let’s take a closer look at each of these components:

Tables : In DynamoDB, data is organized into tables, which serve as containers for storing and managing related information. Tables consist of a collection of items and have a primary key that uniquely identifies each item within the table. Tables are schema-less, meaning that each item in a table can have a different set of attributes.
Items : Items represent individual records within a DynamoDB table. Each item is a collection of attributes, which can vary in number and type between different items in the same table. Items are analogous to rows in a traditional relational database but provide more flexibility since they do not require a fixed schema.
Attributes : Attributes are the key-value pairs that make up the data stored within DynamoDB. Each item can have one or more attributes, where the attribute name represents the key, and the attribute value represents the corresponding value. DynamoDB supports different attribute types, including numbers, strings, binary data, sets, and more.

DynamoDB Tables: The Foundation of Data Storage

In Amazon DynamoDB, tables serve as the foundation for storing and managing your data. They act as containers that organize and structure your information. Understanding DynamoDB tables is essential for effectively working with the service and building scalable and high-performing applications. In this section, we’ll explore DynamoDB tables in detail and discuss their role in data storage.

Key Characteristics of DynamoDB Tables:

Schema-less Nature : Unlike traditional relational databases, DynamoDB tables are schema-less. This means that each item within a table can have a different set of attributes, offering flexibility in data modelling. You can easily add, remove, or modify attributes for different items without altering the table’s structure.
Primary Key : Every DynamoDB table has a primary key that uniquely identifies each item within the table. The primary key can be of two types:

Partition Key (Hash Key): A single attribute that determines the partition in which an item is stored. DynamoDB uses the partition key value to distribute data across multiple storage nodes for scalability and performance. Partition keys should have a high cardinality to evenly distribute the workload.
Composite Primary Key (Partition Key + Sort Key): In addition to the partition key, a sort key (also known as the range key) allows for the efficient querying and sorting of items within a table. The combination of the partition key and sort key creates a unique identifier for each item.

Data Distribution and Scalability : DynamoDB automatically distributes data across multiple partitions based on the partition key to achieve high scalability and performance. As data grows, DynamoDB transparently manages the distribution of data across partitions, allowing your applications to handle increasing workloads without manual intervention.
Data Replication : DynamoDB automatically replicates data across multiple Availability Zones within a region to ensure high availability and durability. This replication strategy provides fault tolerance and protects against data loss in the event of a failure in one Availability Zone.
Flexible Capacity : DynamoDB offers flexible capacity management. You can specify the desired provisioned throughput capacity (measured in read and write capacity units) when creating a table. Provisioned capacity ensures that your application can handle the expected workload. Additionally, DynamoDB can automatically scale your table’s capacity up or down based on demand using auto scaling, providing cost optimization and elasticity.
Global Tables : For globally distributed applications, DynamoDB offers Global Tables, which replicate tables across multiple AWS regions. Global Tables enable low-latency access to data from any region and provide redundancy and disaster recovery capabilities.

Designing DynamoDB Tables:

When designing DynamoDB tables, it’s crucial to consider your application’s access patterns, query requirements, and data relationships. Effective table design can significantly impact query performance and cost efficiency. Factors to consider include choosing appropriate attribute types, defining primary key structures, and leveraging secondary indexes to support various access patterns.

DynamoDB provides flexible and scalable data storage, allowing you to focus on developing your applications without worrying about managing underlying infrastructure. By leveraging DynamoDB’s powerful table capabilities, you can build robust, high-performance, and scalable solutions to meet your data storage and retrieval needs.

Items: Individual Records within DynamoDB Tables

In Amazon DynamoDB, items represent individual records within a table. They are the fundamental units of data storage and retrieval. Understanding how items are structured and how they relate to tables is essential for effectively working with DynamoDB. In this section, we’ll explore items in detail and discuss their role in storing and retrieving data.

Key Characteristics of DynamoDB Items:

Attribute-Value Pairs : Each item in a DynamoDB table consists of one or more attribute-value pairs. Attributes represent the keys, and their corresponding values represent the data associated with those keys. Attributes can be of various data types, such as strings, numbers, binary data, sets, or documents, offering flexibility in data modeling.
Schema-less Nature : DynamoDB’s schema-less nature extends to items as well. Each item within a table can have a different set of attributes. This allows you to store and retrieve items with varying attributes, without the need for a fixed table schema. As your application evolves, you can add or remove attributes from items without impacting other items in the table.
Primary Key : Every item in a DynamoDB table is uniquely identified by its primary key. The primary key can be either a partition key (also known as a hash key) or a composite key consisting of a partition key and a sort key (also known as a range key). The primary key ensures the uniqueness and efficient retrieval of items within the table.
Accessing Items : You can access items in DynamoDB using the primary key. Retrieving an item by its primary key is an efficient operation, providing fast and predictable access to data. DynamoDB supports both single-item retrieval and batch retrieval for multiple items.
Flexible Data Modeling: DynamoDB’s flexible data model allows you to store heterogeneous items within the same table. Items can have different attributes, allowing you to represent diverse data structures. This flexibility enables you to adapt to evolving application requirements and easily incorporate new attributes or data types.

Designing DynamoDB Items:

When designing items in DynamoDB, it’s important to consider the access patterns and query requirements of your application. The attributes you choose and the way you structure your items can significantly impact the efficiency of data retrieval operations. It’s often beneficial to denormalize your data and include all necessary attributes within an item to minimize the need for additional queries.

Additionally, you can leverage DynamoDB’s support for secondary indexes to enhance query flexibility. Secondary indexes allow you to define additional attributes as alternate keys for querying items based on different access patterns.

By effectively structuring and modeling your items, you can optimize data retrieval, ensure scalability, and minimize the need for complex joins or multiple round-trip operations.

Attributes: Key-Value Pairs of Data

In Amazon DynamoDB, attributes are the fundamental components that make up the data stored within items. They are represented as key-value pairs, where the attribute name acts as the key, and the attribute value represents the corresponding data associated with that key. Attributes play a crucial role in the structure, organization, and retrieval of data in DynamoDB. Let’s explore attributes in more detail.

Key Characteristics of DynamoDB Attributes:

Key-Value Structure : Attributes in DynamoDB follow a key-value structure, where the attribute name serves as the key and the attribute value holds the corresponding data. This structure allows for flexible and dynamic data modeling since the attributes associated with an item can vary in number and type.
Attribute Names: Attribute names are used to identify and access specific pieces of data within an item. Attribute names must be unique within an item and should follow the naming rules defined by DynamoDB, such as being case-sensitive and avoiding reserved words.
Attribute Values : Attribute values hold the actual data associated with the attribute name. DynamoDB supports various data types for attribute values, including strings, numbers, binary data, sets, documents, and more. The choice of data type depends on the nature of the data being stored and the desired operations to be performed on that data.
Flexible Data Modeling : DynamoDB’s flexible data model allows you to include different attributes within an item. Each item can have its own set of attributes, providing a schema-less approach. This flexibility allows you to add or remove attributes dynamically, adapting to changing requirements without altering the table’s structure.
Nested Attributes : DynamoDB also supports nested attributes or complex data structures within items. You can represent hierarchical data by using attribute names separated by dots (e.g., “address.city”). This enables you to store and retrieve nested data structures within a single item, simplifying data representation and retrieval.
Attribute Size Limitations : DynamoDB imposes certain size limitations on attribute names and values. The maximum size for an attribute name is 255 bytes, while the maximum size for an attribute value varies depending on the data type. It’s important to consider these limitations and ensure that your data fits within the allowed boundaries.

Attributes in DynamoDB provide a versatile and dynamic approach to storing and accessing data. By leveraging the key-value structure and flexible data modeling capabilities, you can design efficient data models and adapt to changing requirements easily.

Conclusion:

In this comprehensive guide, we’ve delved into the essential components of the DynamoDB data model, understanding how tables, items, and attributes work together to form a robust and flexible NoSQL database. We’ve explored the key concepts behind each element and how they influence data organization, retrieval, and scalability in DynamoDB.

Tables serve as the foundation of data storage, and in upcoming post we will learn how to design effective table structures by defining primary keys, choosing the right attributes, and considering the benefits of Local Secondary Indexes (LSIs) and Global Secondary Indexes (GSIs). With this knowledge, you can create well-optimized DynamoDB tables that cater to diverse access patterns and query requirements.

Items represent individual records within tables, and we’ve discussed their uniqueness, size considerations, and the significance of primary keys in locating specific items efficiently. By mastering items, you can ensure quick and precise data retrieval, critical for delivering a responsive user experience in your applications.

Attributes are the key-value pairs that define the data within items, and we’ve explored various attribute types, including scalar types, set types, and nested attributes. Understanding attribute types is fundamental to modeling your data effectively and utilizing DynamoDB’s full capabilities.

As you embark on your journey with DynamoDB, we encourage you to dive deeper into its vast potential. DynamoDB is more than just a database; it’s a game-changer for modern application development. By harnessing its power, you can build scalable, high-performance applications that cater to the ever-changing demands of your users.

For the latest insights, expert tips, and practical examples on working with DynamoDB and other AWS services, we invite you to join our growing community at AWSMAG. By signing up today, you’ll gain access to exclusive content, in-depth tutorials, and stay updated with the latest trends and advancements in the AWS ecosystem.

Originally published at https://awsmag.com on July 16, 2023.

Getting Started with Amazon DynamoDB: A Beginner’s Guide

Simranjeet Singh — Sat, 01 Jul 2023 07:10:08 +0000

Amazon DynamoDB is a fully managed NoSQL database service provided by Amazon Web Services (AWS). It is designed to provide fast and predictable performance with seamless scalability for applications that require low-latency, high-performance data storage. DynamoDB is built to handle massive workloads and is capable of scaling horizontally across multiple servers to accommodate growing data volumes and traffic.

DynamoDB offers a flexible, key-value store model where data is organized into tables. Each table consists of multiple items, and each item is a collection of attributes. Unlike traditional relational databases, DynamoDB does not require a fixed schema, allowing for dynamic and agile data modeling. This flexibility makes it well-suited for applications with evolving data requirements.

Key features of Amazon DynamoDB include:

Scalability : DynamoDB automatically scales its storage and throughput capacity to accommodate the workload demands. It can handle millions of requests per second and scales up or down seamlessly to meet changing traffic patterns.
Performance : DynamoDB provides low-latency, single-digit millisecond response times, making it ideal for applications that require real-time data access and fast query performance.
Fully Managed : AWS handles the operational aspects of DynamoDB, such as hardware provisioning, software patching, and infrastructure management. This allows developers to focus on building applications without worrying about the underlying infrastructure.
Durability and Availability : DynamoDB replicates data across multiple Availability Zones within a region to ensure high availability and durability. It provides built-in fault tolerance, automatic data backups, and multi-region replication capabilities for global deployments.
Rich Querying : DynamoDB supports fast and efficient querying using primary keys and secondary indexes. It offers various querying options, including key-value access, range queries, and filtering capabilities.
Built-in Security : DynamoDB offers robust security features, including fine-grained access control using AWS Identity and Access Management (IAM), encryption at rest with AWS Key Management Service (KMS), and network isolation using Amazon Virtual Private Cloud (VPC).
DynamoDB Streams : This feature captures and provides a time-ordered sequence of item-level modifications in a DynamoDB table. Streams enable real-time data processing, replication to other services, and event-driven architectures.

Amazon DynamoDB is commonly used for a wide range of applications, such as e-commerce, gaming, mobile, ad tech, IoT, and more. Its flexible scalability, high performance, and serverless architecture make it a popular choice for developers who want to focus on building their applications while offloading the burden of managing databases at scale.

Overall, DynamoDB provides a reliable and feature-rich NoSQL database solution in the AWS ecosystem, allowing developers to build scalable and responsive applications with ease.

DynamoDB’s Data Model

When working with Amazon DynamoDB, it’s essential to understand the data model it employs. DynamoDB, a fully managed NoSQL database service offered by Amazon Web Services (AWS), uses a schema-less and flexible data model, allowing for rapid and scalable application development. In this section, we’ll provide an introduction to DynamoDB’s data model, exploring the key concepts and components that form its foundation.

At the core of DynamoDB’s data model are three key elements: tables, items, and attributes. Let’s take a closer look at each of these components:

Tables : In DynamoDB, data is organized into tables, which serve as containers for storing and managing related information. Tables consist of a collection of items and have a primary key that uniquely identifies each item within the table. Tables are schema-less, meaning that each item in a table can have a different set of attributes.
Items : Items represent individual records within a DynamoDB table. Each item is a collection of attributes, which can vary in number and type between different items in the same table. Items are analogous to rows in a traditional relational database but provide more flexibility since they do not require a fixed schema.
Attributes : Attributes are the key-value pairs that make up the data stored within DynamoDB. Each item can have one or more attributes, where the attribute name represents the key, and the attribute value represents the corresponding value. DynamoDB supports different attribute types, including numbers, strings, binary data, sets, and more.

To uniquely identify items within a table, DynamoDB uses a primary key. The primary key can be of two types:

Partition Key : Also known as the hash key, it is a single attribute that DynamoDB uses to distribute data across multiple partitions for scalability and performance.
Composite Primary Key : In addition to the partition key, a composite primary key includes a sort key (also known as the range key). The combination of the partition key and sort key allows for efficient querying and sorting of items within a table.

By leveraging primary keys and secondary indexes, DynamoDB provides powerful querying capabilities. Secondary indexes allow you to define additional attributes to support different access patterns, enhancing query flexibility and performance.

Understanding DynamoDB’s data model is crucial for effectively designing and working with your database. It empowers you to make informed decisions about table structures, key designs, and access patterns to optimize performance and scalability for your applications.

In the upcoming sections of this blog series, we will delve deeper into each of these components, explore advanced data modeling techniques, and provide best practices to help you make the most out of DynamoDB.

Stay tuned for our next article, where we’ll discuss DynamoDB tables and their role in organizing data. Subscribe to our blog to receive updates and never miss a post!

Originally published at https://awsmag.com on July 1, 2023.

Amazon API Gateway HTTP Errors

Simranjeet Singh — Mon, 24 Jan 2022 06:07:00 +0000

Amazon API Gateway is a fully managed service that helps developers to create and deploy scalable APIs on AWS. These APIs act as an entry point for the applications to connect and get access to data, perform business logic or access any other AWS service.

Amazon API Gateway also returns some HTTP Errors and we will be discussing some of the errors in this blog post and what they mean when returned from the Amazon API Gateway. Usually, the errors returned are in the range of 4xx or 5xx and examples for the same are 400 or 500. As a rule of thumb errors in the range of 400–499 are usually returned if there are problems with the client or you are breaking some of the rules defined by the Amazon API Gateway.

The errors in the range of 500–599 mean the server is not working problem or you have an issue with the network or the issues in the infrastructure which runs your server.

400 Error: Bad Request

The HTTP Status 400: Bad Request is the broadest error and depending on what AWS service API Gateway is integrating with, this error means many things. Some of the reasons for this error can be an invalid JSON, wrong data types and required fields etc.

403 Error: Access Denied

The HTTP Status 403: Forbidden means there are some permission issues. In AWS, this can be an issue with a wrong IAM role configuration. If your service uses an auth mechanism like AWS Cognito or a custom authorizer, this can be a permission issue because of this, then this error code will be returned.

404 Error: Not Found

The HTTP Status 404: This means the resource is not available or the URL does not exist. You can check the URL if it is right or not or have been implemented right to make sure that you are not making any mistake.

409 Error: Conflict

The HTTP Status 409: indicates that your request is trying to do something that conflicts with the current state of the target resource. It is most likely to occur in response to a PUT request.

429 Error: Too Many Requests

There are two cases when you can receive 429 errors from API Gateway.

The first one for HTTP Status 429: “Too Many Request”. This usually happens when the downstream resource is not able to handle the number of requests coming in.

For example, if you have a Lambda which gets triggered via an API Gateway and there is a reserved concurrency assigned to it let’s say 20 then 21 requests will same time probably give you this error.

This can also happen if your API keys are not allowing more than x number of requests concurrently. If the number of requests exceeds the number even if the downstream resource can handle it, the API Gateway will give this error.

429 Error: Limit Exceeded

The second one for HTTP Status 429 is “Limit Exceeded Exception,” which means that you have exceeded the allowed number of requests. This happens when the request is metered using an API key in API Gateway. The usage plan is associated with the key and the plan decides how many requests are allowed in a month by that particular resource.

500 Error: Internal Server Error

HTTP Status 500: It is the most generic HTTP error you will see. If the downstream service is Lambda, this error can mean an issue or a bug in the code of the function.

This can also happen if the status code mapping in the API is wrong. The default mapping if the error mapping is not configured properly, then the status code returned to the client is HTTP Status code 500.

502 Error: Bad Gateway

HTTP Status 502: this usually happens when the downstream service is not able to provide a response that can be mapped easily with the API Gateway. Sometimes the downstream service is not ready and cannot return a response.

Amazon API Gateway has a hard limit of 30 seconds timeouts. If the downstream service is not able to respond in this time frame, the API Gateway returns the HTTP Status 503.

Conclusion:

The above-mentioned codes are some of the common errors which you may encounter while working with the Amazon API Gateway. If you would like to be familiar with the other things related to Amazon API Gateway, we have a collection that lists all the articles related to it. You can find it here.

Originally published at https://awsmag.com on January 24, 2022.

What is Amazon Cognito?

Simranjeet Singh — Tue, 04 Jan 2022 12:43:15 +0000

Mobile or web apps mostly requires a user management solution to manage, authenticate users before giving them access to a restricted area in the app. Creating a user management system from scratch is a big task and requires a deep understanding of handling the PII data i.e. personally identifiable information. Amazon Cognito provides this solution to the developers. Using Amazon Cognito, you can manage and authenticate your users before giving them access to the restricted area in your app.

Amazon Cognito provides authentication, authorization, and user management as a service for your web and mobile apps. It allows users to create an account using a username or password, have a configured MFA (Multi-Factor Auth) and also enable them to log in using one of the third-party providers like Facebook and Google.

User Pools are user directories in Amazon Cognito which provide sign-up and sign-in options for your users. Users can create their accounts and use the credentials to log in to your application.
Identity Pools allows you to grant access to your users so that they can access other AWS services.

Concepts in Amazon Cognito

The two main concepts of Amazon Cognito are:-

User Pools are user directories in Amazon Cognito which provide sign-up and sign-in options for your users. Users can create their accounts and use the credentials to log in to your application.
Identity Pools allows you to grant access to your users so that they can access other AWS services.

What are user pools in Cognito?

Your users can Sign-up and Sign-in to your app.
Amazon Cognito has a built-in, customizable web UI to Sign-in to your app.
You can use social Sign-in with Facebook, Google, Login with Amazon, and Sign in with Apple with your user pool.
There is also an option for Sign-in through SAML and OIDC identity providers.
User pools are user directories with features that can help you manage your users and their profiles easily.
User Pools from Amazon Cognito have security features like multi-factor authentication (MFA), checks for compromised credentials, account takeover protection, and phone and email verification.
Customized workflows for user Sign-up and Sign-in are also available through AWS Lambda triggers.

User Pools are user directories in Amazon Cognito which provide sign-up and sign-in options for your users. Users can create their accounts and use the credentials to log in to your application. It also allows your users to federate through a third-party identity provider (IdP) like Facebook or Google. Whether your users use the password-based account or use a third party to create an account with your app, all the users will be created as a member of the user pool and every member will have a directory profile that you can access through an SDK.

Following are the features of a User Pool:

User pools along with identity pools allow your application to federate using a third-party provider and save the information in your user directory. With the use of the Identity Pools, you can also grant temporary access to AWS services like S3 or Dynamodb.

Integration With Cognito

Many AWS services like API Gateway directly integrate with Amazon Cognito user pools to authenticate the API request to the gateway. AWS Amplify is another library that provides auth setup using Amazon Cognito. it is easy to spin up an auth service using AWS Amplify and many new age apps are using it.

If you are interested to check out how can we use Amazon API Gateway and Amazon Cognito user pool, you can read it here.

Conclusion

Amazon Cognito is a cost-effective, secure and highly scalable Authentication service. If you are not worried about the vendor lock or most of your infrastructure is deployed on AWS, you can give it a try.

Originally published at https://awsmag.com on January 4, 2022.

How to create a VPC using Terraform?

Simranjeet Singh — Sun, 01 Aug 2021 07:25:50 +0000

Running your applications comes up with other challenges too and one of those challenges is having a robust network set up to host all parts in one place. We have to set up VPC (Virtual Private Cloud), internet gateway, subnet, etc. to make sure our application is working properly. The other aspect of this is to manage the infrastructure once it is ready and deployed. This is where Terraform comes in handy. Terraform is an Infrastructure-as-a-code that helps you to define infrastructure in code and you can easily maintain it for future updates.

If you are not aware of the networking fundamentals on AWS, read the article AWS Networking Fundamentals before going deep with Terraform in this article.

You need to have some information about how Terraform works. If you don’t know about Terraform, I suggest going through its documentation to have a basic idea about it before diving into the article.

I will post the snippets and add some description in steps here. You can also find the complete module at a GitHub repo aws-vpc-terraform .

Directory Structure

The above directory structure of the module has the following key files:

main : Contains the entire module and all the resources we will discuss in some time.
output : Defines the output provided by the module. This provider returns the vpcId of the VPC created by the module.
provider: Defines the provider required for the module to work properly. You can also think of it as dependencies required for the module. This module needs the AWS module from Hashicorp (creators of Terraform). The AWS module will allow us to use the resources available in the AWS to create our desired infrastructure.
variables: It contains the input variables required by the module to complete its task. For the sale of this article, I have set default values to the variables but they can be easily made required by removing the default value.

As mentioned above the most important file is main.tf which contains all the code for the resources we are about to create. Let’s go through each resource statement in file and understand them a bit.

data "aws_availability_zones" "availableAZ" {}

# VPC 
resource "aws_vpc" "vpc" { 
  cidr_block = var.cidr 
  instance_tenancy = "default" 
  enable_dns_support = true 
  enable_dns_hostnames = true 
  assign_generated_ipv6_cidr_block = true 
  tags = { 
    Name = var.namespace 
    Namespace = var.namespace 
  } 
}

# Public Subnet 
resource "aws_subnet" "publicsubnet" { 
  count = 3 
  cidr_block = tolist(var.publicSubnetCIDR)[count.index] 
  vpc_id = aws_vpc.vpc.id 
  map_public_ip_on_launch = true 
  availability_zone = data.aws_availability_zones.availableAZ.names[count.index] 
  tags = { 
    Name = "${var.namespace}-publicsubnet-${count.index + 1}" 
   AZ = data.aws_availability_zones.availableAZ.names[count.index]
   Namespace = var.namespace 
  } 
  depends_on = [aws_vpc.vpc] 
}

# Private Subnet 
resource "aws_subnet" "privatesubnet" { 
  count = 3 
  cidr_block = tolist(var.privateSubnetCIDR)[count.index] 
  vpc_id = aws_vpc.vpc.id 
  availability_zone = data.aws_availability_zones.availableAZ.names[count.index]
  tags = { 
    Name = "${var.namespace}-privatesubnet-${count.index + 1}"
    AZ = data.aws_availability_zones.availableAZ.names[count.index]
    Namespace = var.namespace 
  }
  depends_on = [aws_vpc.vpc] 
}

# Internet Gateway 
resource "aws_internet_gateway" "internetgateway" { 
  vpc_id = aws_vpc.vpc.id 
  tags = { 
    Name = "${var.namespace}-InternetGateway" 
    Namespace = var.namespace 
  }
  depends_on = [aws_vpc.vpc]
}

# Elastic IP 
resource "aws_eip" "elasticIPs" { 
  count = 3 
  vpc = true 
  tags = { 
    Name = "elasticIP-${count.index + 1}"
    Namespace = var.namespace 
  } 
  depends_on = [aws_internet_gateway.internetgateway] 
} 

# NAT Gateway 
resource "aws_nat_gateway" "natgateway" { 
  count = 3 
  allocation_id = aws_eip.elasticIPs[count.index].id 
  subnet_id = aws_subnet.publicsubnet[count.index].id 
  tags = { 
    Name = "${var.namespace}-NATGateway-${count.index + 1}"
    AZ = data.aws_availability_zones.availableAZ.names[count.index]
    Namespace = var.namespace 
  } 
  depends_on = [aws_internet_gateway.internetgateway] 
}

We have all the major parts of the network and now it is time to create route tables. Route Tables define which traffic can flow to which resource. We will create a Route Table for public and private subnets.
Public Route Table will have the traffic flowing from Internet Gateway directly. We will also create an association record to associate the newly created route table with the public subnets.

# Route Table for Public Routes 
resource "aws_route_table" "publicroutetable" { 
  vpc_id = aws_vpc.vpc.id 
  route { 
    cidr_block = "0.0.0.0/0" 
    gateway_id = aws_internet_gateway.internetgateway.id 
  } 
  tags = { 
     Name = "${var.namespace}-publicroutetable" 
     Namespace = var.namespace 
  } 
  depends_on = [aws_internet_gateway.internetgateway] 
} 

# Route Table Association - Public Routes 
resource "aws_route_table_association" "routeTableAssociationPublicRoute" { 
  count = 3 
  route_table_id = aws_route_table.publicroutetable.id 
  subnet_id = aws_subnet.publicsubnet[count.index].id 
  depends_on = [aws_subnet.publicsubnet, aws_route_table.publicroutetable]
}

# Route Table for Private Routes 
resource "aws_route_table" "privateroutetable" { 
  count = 3 
  vpc_id = aws_vpc.vpc.id 
  route { 
    cidr_block = "0.0.0.0/0" 
    gateway_id = aws_nat_gateway.natgateway[count.index].id 
  } 
  tags = { 
    Name = "${var.namespace}-privateroutetable-${count.index + 1}"
    AZ = data.aws_availability_zones.availableAZ.names[count.index]
    Namespace = var.namespace 
  } 
  depends_on = [aws_nat_gateway.natgateway] 
} 

# Route Table Association - Private Routes 
resource "aws_route_table_association" "routeTableAssociationPrivateRoute" { 
  count = 3 
  route_table_id = aws_route_table.privateroutetable[count.index].id
  subnet_id = aws_subnet.privatesubnet[count.index].id 
  depends_on = [aws_subnet.privatesubnet, aws_route_table.privateroutetable] 
}

Our entire module is ready. In order to run it, you have to first initialize the Terraform, see the plan and apply it to create your VPC using Terraform.

terraform init 
terraform plan 
terraform apply

That’s all related to deploying and managing your VPC using Terraform. In upcoming articles, I will write more about creating other services and deploying some common things using Terraform. Till then Happy Coding.

Originally published at https://awsmag.com on August 1, 2021.

What is AWS SAM (Serverless Application Model)?

Simranjeet Singh — Thu, 08 Jul 2021 11:03:00 +0000

AWS SAM (Serverless Application Model) is an open-source framework to develop and deploy serverless applications on AWS. The serverless application in the case of AWS is a combination of Amazon Lambda, databases, Amazon API Gateway etc. If you like to read more about serverless computing before diving deep into the AWS SAM, you can read it here.

What is a Serverless Application?

Before we understand AWS SAM, let us first understand what is a serverless application.

A Serverless Application is a combination of various serverless services provided by AWS. These services are Amazon Lambda, Amazon API Gateway etc. All these services work together with each other and form an application to serve your customers.

Benefits of AWS SAM

Some of the benefits of AWS SAM are:

Testing & Debugging : We discussed in the serverless computing post that it is hard to test and debug serverless applications. To solve this AWS SAM provides an AWS Lambda like local execution environment which you can use to run and debug the functions locally.
CloudFormation Support : AWS Sam is built on CloudFormation and supports all the resources of Cloudformation in its config file. The AWS SAM team have also created some special construct which you can use to create and deploy resources with less amount of code. For example, A WS::Serverless::Function allows you to create an AWS Lambda function along with all the required events configured in the config. If you have configured an API gateway event for your AWS Lambda function, the AWS SAM will deploy that API gateway as well.
Single Deployment Configuration: As I mentionedthat AWS SAM can deploy resources using a construct and also supports other CloudFormation resources as well, you will only need a single deployment config to define and deploy your resources on AWS.
Integration : AWS SAM can also integrate with your CI/CD tool of choice to automate your deployment pipeline. In AWS, AWS Cloud9 IDE supports AWS SAM so you can author, test serverless applications.

That was a quick introduction to AWS SAM. I will be adding more posts around getting started with AWS SAM and building applications using AWS SAM.

Originally published at https://awsmag.com on July 8, 2021.

AWS Networking Fundamentals

Simranjeet Singh — Fri, 24 Jul 2020 03:51:38 +0000

PS: The post was originally posted on my weekly AWS newsletter - AWSMAG. If you wish to receive more like these every week, join the newsletter.

When I started using AWS as a developer, I was bombarded with the lots of jargon about VPC, subnet, CIDR ranges and all those words, I was not able to remember in the beginning. If this is something happened to you as well, you are in the right place my friend. Let's understand the fundamentals of AWS networking in this blog post.

First of all, we will be using some words like Region, Availability zones etc. in this and if you are not familiar with these words, you can read my other post about AWS Global Architecture. Once you have an idea what these are, we can move ahead with other jargons.

So, what are we going to talk about here. Following is the List:

VPC(Virtual private cloud)
Subnet
Security Groups
Internet Gateway
NAT Gateway
CIDR Range

Following Image will give you an idea how all this fits in.

What is VPC (Virtual Private Cloud)?

VPC or Virtual Private Cloud is a logical isolation in AWS cloud which is defined by you for creating your own
infrastructure in it. Consider this as your own space.
You can create your own network and configure it in they way you like it to work. All the resources are deployed in it and they are isolated from the resources deployed in any other VPC. From a hierarchical point of view, you have a Region and you will create a VPC in it which will hold all of your resources. By default, every Region comes up with a default VPC which is a good starting point and you can use it if you don't want to get your hands dirty. My advice will be to get your hands dirty and create a network which suits your requirements. Default VPC is already configured with the following:

A CIDR Range
Subnets to access private and public network
Internet and NAT Gateway
Security Groups

If we want to create our own VPC, we will need to understand and create all the things mentioned above. so lets try to understand them first.

What is a CIDR Range?

A CIDR(Classless Inter-Domain Routing) range is a group of IP addresses which you can use in your network. As it is a private network, we use IP Address from RFC1918 standard. These addresses are not used over internet, so it is safe to use them internal even though when we connect our network over internet. Also, one thing we should keep in mind is
if we are connecting to different VPC, we should take care that both of them are not using same private address range. Otherwise we will have issues while connecting them. Let's look at a CIDR Range.

The first part is network, second part defines the host and the part after / tells you how many addresses can be utilised. This will have 65,500 addresses approx.

Usually applications are deployed in different Availability Zones to maintain high availability and have redundancy.
This allows us to handle any failover situations. To reduce the blast radius and to use of this offering by AWS, we divide CIDR range in equal parts to all availability zones.
For eg: -

AZ-a will have 172.31.0.0/24 - this means it has approx 250 address.
AZ-a will have 172.31.1.0/24 - this means it has approx 250 address.
AZ-a will have 172.31.2.0/24 - this means it has approx 250 address.

After the division you will still have addresses left for your rest of the structure. Now we have created and divided the addresses, how will they talk to each other.
That is where subnet comes in.

What is a Subnet?

The division we mentioned above i.e. the division of the Ip addresses in multiple AZ's is actually using subnet. Subnet is a logical group of Ip addresses
that is a subsection of the wider network which we talked earlier. Subnets are of two types and here they are:-

Private Subnet: A private subnet is a group of addresses which you don't want anyone can access from outside the VPC. We usually use private subnet for things like database. We want to keep it out of the hands of anything which can access over the internet. But, any application deployed in our network should be able to access it. So any application you have deployed internally which needs to process data stored in your database should be able to access it. Now, when I came to know about this in my early days, I asked a question. What if I want to apply a patch on my database system which is released on internet? If you also have that question in your mind, well the answer is NAT Gateway. and we will talk about them after some time.
Public Subnet: Yes, you guessed it right, A public subnet is a logic group of addresses which we want to access via internet. These are a good example for your web servers. If you are hosting an application which is web based and you want to access it, we will assign them in our public subnet and they should be able to get request and also call things over internet with the help of an Internet Gateway. We will talk about it in a moment.

Each Subnet is assigned a route table which will tell you what it can access. The route table holds rules which allow them to access VPC resources or internet via an
Internet Gateway. You can provide CIDR ranges in here to define the flow of the network. something like 0.0.0.0/0 means all traffic goes to that ip address range. We often use
this to setup Internet Gateway.

Subnets are a very important piece in your architecture. They tell you what type of network access an instance has and in which Availability Zone the instance lies in.
They also have a feature called Network Access Control List(ACLs) which is a security feature and tells you which IP addresses and ports are allowed to flow traffic
in and out in the subnet.

Ok, you made it so far. Do you need a cup of coffee before we move ahead? if Yes take a break. If no, let's tackle three big words we came across while discussing Subnets. Let's Start with Internet Gateway.

What is Internet Gateway?

Internet Gateway is a component which provides internet accessibility to your VPC. Why do we need it? Because we want someone from the internet should be able to access our resources. Like I mentioned above, this is a usual case for a web server. If you have a website hosted you want people to access it over internet.
They can only do it when you allow it on your network. Getting an Internet Gateway does not solve your problem. You also need to connect your public subnet to this gateway by adding an entry in the route table like 0.0.0.0/0. This allows all the traffic to flow from the Internet Gateway providing you the access over internet.

What is NAT Gateway?

NAT Gateway is managed service by AWS which lets you connect to the internet. Now, you will be like, Dude!! then what the hell Internet Gateway is used for? Well they both provide access to internet but NAT gateway is used with your private subnet and technically goes via your internet gateway. Any instance in the private subnet should be able to go to internet to access any patches or downloading dependencies.
Someone on the internet should not be able to talk to our private instances. Thats where NAT gateway helps us. Initially we used to deploy NAT instances to achieve the same functionality.

We discussed the whole networking and getting our infra right. Deploying our instances on different AZ for better availability and redundancy and everything. The most important piece in all this is security. How will we maintain security across all this? The answer is Security Groups.

What is a Security Group?

Security Groups are virtual firewall around our instances. Do not get confuse with Network Access List which we talked about in Subnets. ACL's are at subnet level and control the traffic for all the instances in that subnet. Security groups are at instance level and controls the flow of data at instance level. One example will be, you can create a security group for your database service and application service.
Database service should be able to access database in the private subnet on a particular port but this flexibility will not be allowed in the application service security group.
Any application service under the security group will have access to database service but not to database directly. So they have to go via the service to talk to database. Another example can be to not allow ssh access to your instance on any port to allow it for one IP range only which will be your enterprise range.

That's all the fundamental jargon you should know if you are working with the AWS and developing application deployed on it. Understanding these is also important as the defined rules and constraints over the network also affect the decisions we take while developing the applications. Else we will be in a position when some people say, It works on my local, but blows up when I deploy.
Don't be like that.

Have a nice cup of coffee and digest all these. In future, I will write about some advance things related to the AWS networking.

AWS Global Infrastructure

Simranjeet Singh — Tue, 07 Jul 2020 04:43:39 +0000

Ever wondered how AWS provide robust infrastructure and high availability to our apps running in cloud. Well in this blog post, we will
answer the question. And the Answer is AWS Global Infrastructure. AWS maintains a global infrastructure which is uses to give you the
peace of mind that my app is secure and is running in high availability. This does not mean it is by default as you have to configure it for
high availability. So let's look into the parts of the AWS Global infrastructure and try to understand what it is.

The AWS Global Infrastructure consists of Regions, Availability Zones, Local Zones, Point of Presence etc. Let's understand the heavy words we came across above. Have a look at the following image:-

In the above Image, we have region as the starting point. Everything is wrapped inside the region. So what is a Region?

Region: Region is an actual geographic location maintained by AWS to provide a global footprint. Every Region provides
full redundancy and connectivity to the network. All the regions in the AWS are connected using a global network so one region
can talk to another region. Every region consistence of multiple Availability Zones which are different data centers to provide redundancy for
your application. Lets look at the naming convention of the regions: eu-west-1 and eu-west-2. Here "eu" means that it is in European continent,
"west" means it is on the west side and "1" is the number of the aws region in that area. eu-west-2 is second aws region on west side of European
continent. All the regions also have their city names where they are located. eu-west-1 is in Ireland and eu-west-2 is in London.
There are special regions which have Gov cloud in their name. This is because these regions are utilised by govt. and are not used by any other clients of AWS.

Availability Zones: We talked about Availability Zones when we were defining regions. It is a fully isolated partition in AWS infrastructure.
What does this mean? It means every zone is a physicaaly different location using separate connections, power supply etc. This isolation gurantees
the redundancy and high availability for your applications. So if due to some reason, one availability zone goes down, the application hosted on the other one will
be able to serve your customers. You have to configure your deployments to use availability zones. Some of the AWS services provide this by default
like SQS. Every Availbility Zone consists of multiple data centers.

Point of Presence: These locations are consists of Edge Locations and Regional Cache Servers. There is a CDN service by Amazon called
Amazon Cloudfront, uses the location to cater customers globally with low latency and high transfer speeds. Another service launched by AWS called Global
Accelerator also use these to upgrade the performance of your application.

Local Zones: Local Zones are new offering in AWS. These are extensions of the Regions in geographic proximity to users. Every Local
Zone has their own internet connection and support AWS Direct Connect. Local Zones are not available in every region and they have to be
enabled in order to be utilised. The naming convention of the Local Zone is "us-west-2-laz-1a". Here us-west-2 is the name of the Region and
lax-1a indicates the location.

That's a higher level picture of AWS Global infrastructure and we will talk more about them when we start looking into other AWS Services.

PS: The post was originally posted on my newsletter and a blog awsmag. If you like to recieve these post and other stuff related to AWS, subscribe to my weekly newsletter at awsmag.

Getting Started With AWS SNS

Simranjeet Singh — Fri, 20 Mar 2020 06:12:15 +0000

Introduction

Amazon SNS (Simple Notification Service) is a fully managed pub/sub messaging service which enables you to write distributed applications. Using SNS, you can manage and send notifications to all the subscribed system using endpoints like SQS and webhooks. It can also send messages to Lambda function further processing. SNS can also be used to maintain large number of human subscribers too. People can get notifications using SMS and emails.

In this part we will see how can we publish message using SNS.

The Flow in SNS.

Before we begin let us first understand what is Publisher/Subscriber model.

Publish/Subscriber Model

There are two components in a system:

Publisher : A service that can broadcast out messages to its subscribers.
Subscriber : Any service that wish to receive the messages broadcasted by the publisher.

In the above diagram the publisher is sending message to a SNS topic via the service and all the subscribers will receive the message but the mode or endpoint using which they have subscribed if different.

Pre-requisites

You will need a valid AWS account and credentials to access the SNS. You will also need to have access to the AWS console to create a SNS topic and some subscribers to it.

Setting up a SNS Topic

To setup a SNS topic you first login to AWS and navigate to SNS. Follow the instruction to create a SNS and a topic. Once created, you will need the ARN property of the SNS to use in the code. Make sure, the credentials you are using have access to publish message from the SNS. Add some subscribers and confirm them to see the full action.

Publishing a message

Let’s assume following is the message.

{
  "foo": "bar"
}

Now that we have a message structure, we need to publish it out to the desired SNS. We have to import the AWS SDK for node.js and use it to publish a message . The SDK is capable of using the credentials stored in your env. It looks for the following environment variable:-

export AWS_ACCESS_KEY_ID=your_access_key_idexport AWS_SECRET_ACCESS_KEY=your_secret_access_keyexport AWS_REGION=the_region_you_are_using

Following is the code to publish the message:-

/* Getting Started with AWS SNS using node js. This part shows how to publish content to SNS */

// Load the AWS SDK for Node.js
const AWS = require("aws-sdk");

const sns = new AWS.SNS({apiVersion: "2010-03-31"});
const params = {
  "Message": JSON.stringify({"foo": "bar"}),
  "TopicArn": "ARN FOR TOPIC YOU WANT TO PUBLISH TO"
};

// By using Callback
sns.publish(params, (err, data) => {
  if (err) {
    console.log("There was an Error: ", err);
  } else {
    console.log("Successfully published.", data);
  }
});

The above is implemented using the callback. if you wish to achieve the implementation using promise, following is the implementation.

// Promise implementation
sns.publish(params).promise()
.then(data => console.log("Successfully published.", data))
.catch(err => console.log("There was an Error: ", err));

You can also find the code sample in my github repo at the following link

singhs020/examples

Conclusion

AWS Simple Notification Service (SNS) is a super scalable service that allows us to implement the publish/subscribe model with ease. We can use this to send texts, emails, push notifications, or other automated messages to multiple channels at the same time. There are many other uses cases and some advanced filtering logic, message templating and mixed message available in SNS. Give it a try and Happy Coding.

Getting Started With AWS SNS

Simranjeet Singh — Fri, 20 Mar 2020 06:10:51 +0000

Introduction

Publish/Subscriber Model

There are two components in a system:

Publisher: A service that can broadcast out messages to its subscribers.
Subscriber: Any service that wish to receive the messages broadcasted by the publisher.

If a service wish to subscribe to a publisher, it needs to notify the publisher that it wants to receive its broadcasts along with where it wishes to receive i.e. the endpoint. It can be Http endpoint, SQS or lambda function.
In the above diagram the publisher is sending message to a SNS topic via the service and all the subscribers will receive the message but the mode or endpoint using which they have subscribed if different.

Pre-requisites

You will need a valid AWS account and credentials to access the SNS. You will also need to have access to the AWS console to create a SNS topic and some subscribers to it.

Setting up a SNS Topic

Publishing a message

Let's assume following is the message.

{
  "foo": "bar"
}

export AWS_ACCESS_KEY_ID=your_access_key_idexport
AWS_SECRET_ACCESS_KEY=your_secret_access_keyexport
AWS_REGION=the_region_you_are_using

Following is the code to publish the message:-

/* Getting Started with AWS SNS using node js. This part shows how to publish content to SNS */


// Load the AWS SDK for Node.js
const AWS = require("aws-sdk");

const sns = new AWS.SNS({apiVersion: "2010-03-31"});
const params = {
  "Message": JSON.stringify({"foo": "bar"}),
  "TopicArn": "ARN FOR TOPIC YOU WANT TO PUBLISH TO"
};

// By using Callback
sns.publish(params, (err, data) => {
  if (err) {
    console.log("There was an Error: ", err);
  } else {
    console.log("Successfully published.", data);
  }
});

The above is implemented using the callback. if you wish to achieve the implementation using promise, following is the implementation.

// Promise implementation
sns.publish(params).promise()
.then(data => console.log("Successfully published.", data))
.catch(err => console.log("There was an Error: ", err));

You can also find the code sample in my github repo

Conclusion

Getting Started with AWS SQS using Node.js - Part 2

Simranjeet Singh — Fri, 13 Mar 2020 05:09:05 +0000

Introduction

In the previous part i.e. Getting Started with AWS SQS using Node.js - Part 1, we had a look on how to send messages in the SQS. You can also call this as the producer of the message.
In this part we will see how can we connect to SQS and receive message for further processing.

Pre-requisites

You should have followed the previous part of the article and can produce messages to a SQS.

The Application Flow

In the previous part we were building an e-commerce app where an order service is producing messages to the SQS for further processing. In this part we will be looking at a fulfilment service which will receive the message and process it further.

Receiving a message

This was the message which was produced in the last part for fulfilment service

{
  "orderId": "this-is-an-order-id",
  "date": "2020–02–02",
  "shipBy": "2020–02–04",
  "foo": "bar"
}

Like we did last time, we have to import the AWS SDK for node.js and use it to send a message . The SDK is capable of using the credentials stored in your env. It looks for the following environment variable:-

export AWS_ACCESS_KEY_ID=your_access_key_idexport
AWS_SECRET_ACCESS_KEY=your_secret_access_keyexport
AWS_REGION=the_region_you_are_using

Following is the code to receive the message:-

/* Getting Started with AWS SQS using node js. This part shows how to consume message from the SQS */


// Load the AWS SDK for Node.js
const AWS = require("aws-sdk");

const sqs = new AWS.SQS({apiVersion: "2012-11-05"});

const qurl = "ADD YOUR SQS URL HERE";

const params = {
  "QueueUrl": qurl,
  "MaxNumberOfMessages": 1
};

sqs.receiveMessage(params, (err, data) => {
  if (err) {
    console.log(err, err.stack);
  } else {
    if (!Array.isArray(data.Messages) || data.Messages.length === 0) { 
      console.log("There are no messages available for processing."); 
      return;
    }    

    const body = JSON.parse(data.Messages[0].Body);
    console.log(body);

    // process the body however you see fit.
    // once the processing of the body is complete, delete the message from the SQS to avoid reprocessing it.

    const delParams = {
      "QueueUrl": qurl,
      "ReceiptHandle": data.Messages[0].ReceiptHandle
    };

    sqs.deleteMessage(delParams, (err, data) => {
      if (err) {
        console.log("There was an error", err);
      } else {
        console.log("Message processed Successfully");
      }
    });
  }
});

Do, not forget to delete the message after you are done with your task. This is important to avoid any re-processing of the message. The above is implemented using callback. if you wish to achieve the implementation using promise, following is the code.

// the above message can be implemented using promise as well.
sqs.receiveMessage(params).promise()
.then(data => {
  console.log(data);
  // do the processing here
});

You can also find the code sample in my github repo.

Conclusion

AWS SQS is a powerful messaging service which allows you to use your own creativity to find the right fit for it in your application. The most common way to consume messages is using a polling mechanism which can poll the SQS and process all the message. This is a very basic integration of SQS in an application, there are other advanced use cases too like dead-letter queues, FIFO queues and Lambda integration with SQS to process streams.