Forem: EightPLabs

RDS on AWS - Step by step guide to creating RDS

EightPLabs — Wed, 06 Mar 2024 11:05:03 +0000

RDS on AWS - Creating RDS

We discussed about pros and cons of RDS and whether to use it or not here

If you made up your mind to use it then this blog covers step by step guide to create and use RDS.

First, you'll need to choose from various database engines like MySQL, PostgreSQL, SQL Server, Oracle, and MariaDB for your RDS instance. Then, select predefined templates suitable for production, development/testing, or utilize the Free Tier RDS setup.

Ensure availability and durability by configuring settings for high availability and data redundancy to guarantee reliable database operation.
Next, adjust configurations related to instance type, storage, networking, and security to meet your application's requirements. You can choose instance types based on workload characteristics, define storage options such as volume type, size, and auto-scaling settings to accommodate data requirements.
Configure network access settings like internal access within AWS services, public access, VPC connectivity, and RDS Proxy setup. Determine connectivity options for EC2 compute resources to interact with the RDS instance, and choose whether to assign a public IP address for external access.
Specify the Virtual Private Cloud (VPC) and associated subnet groups for deploying the RDS instance within a secure network environment. Additionally, create an RDS Proxy for enhanced database scalability, resilience, and security.
Set up authentication methods and define user access permissions using password authentication, IAM roles, or Kerberos authentication. Enable monitoring features like Performance Insights and Enhanced Monitoring to track database performance metrics and diagnose issues.
Configure automated backup settings such as retention periods, backup windows, and replication options for data protection, and enable encryption at rest and in transit to enhance data security and compliance with industry standards.
Schedule automated maintenance tasks such as minor version upgrades and patching to ensure database health and security. Finally, understand the pricing model, estimate costs, and manage billing preferences to optimize spending and budget allocation for RDS usage.

By following these steps, you'll effectively manage your RDS instance on AWS.

Hands-On -Creating RDS

This blog lets us learn how to create an RDS database in the AWS console by following simple steps.

Prerequisites:

These are the prerequisites for creating RDS in the AWS management console

Sign in to the AWS Management Console to access RDS.
Choose a region for deploying your RDS. For this tutorial, we are choosing the Mumbai region.

Step 1: Sign in to the AWS management console, navigate to the RDS dashboard, and click on Create Database.

Step 2: Choose a database creation method.

There are two options available, such as

Standard Create: This method provides full control over all configuration options, allowing customization of availability, security, backups, and maintenance settings according to specific requirements and preferences.
Easy Create: This method offers recommended best-practice configurations for quick deployment, with pre-set settings optimized for common use cases, while still allowing flexibility to change certain options post-creation if needed.

Select a method that is most suitable for your use case. In this guide, Let us choose the default Standard Create method.

Step 3: Select an Engine option

AWS console offers six types of database engines such as

Aurora (MySQL): a High-performance MySQL-compatible database engine.
MySQL: Open-source RDBMS is known for speed and reliability.
MariaDB: MySQL-compatible fork with additional features.
PostgreSQL: Powerful open-source object-relational database system.
Oracle: Commercial RDBMS is known for scalability and security.
Microsoft SQL Server: Commercial RDBMS by Microsoft for data warehousing and BI.
IBM Db2: IBM's RDBMS with advanced analytics and integration capabilities.

Choose an engine that is adaptable for your application. In this guide, Let us select the default _PostgreSQL _database and its default engine version.

Step 4: Choose a template

AWS offers 3 templates for creating RDS. They are

Production

It is a default setting that prioritizes high availability and consistent performance. Production is ideal for deploying databases in a production environment with minimal configuration required.
Dev/Test

It is configured for development purposes outside of production, allowing for experimentation and testing without impacting live systems. This template offers flexibility for developers to customize settings as needed.
Free Tier

Utilizes RDS Free Tier to develop new applications, test existing ones, or gain hands-on experience with Amazon RDS at no cost. This template is suitable for small-scale projects and learning purposes.

For this guide, we will be choosing a Free tier template. In this template, most of the AWS services for optimizing the RDS are restricted.

Step 5: Creating DB instance identifier

DB instance identifier: This is a unique name for your DB instance within your AWS account and region. It must be 1 to 60 alphanumeric characters or hyphens.

Master username: This is the login ID for the master user of your DB instance.

Manage master credentials in AWS Secrets Manager: This option allows you to manage the master user credentials in AWS Secrets Manager. RDS can generate a password for you and manage it throughout its lifecycle.
Create a master password for the master user of your DB instance and confirm it.

Step 6: DB instance configuration.

AWS RDS allows you to select the type of virtual machine (instance) for your database. The available classes depend on the engine you selected. Common options include

Standard classes (includes m classes): Balanced compute and memory resources.
Memory-optimized classes (includes r and x classes): Optimized for memory-intensive workloads.
Burstable classes (includes t classes): Suitable for workloads with occasional bursts of CPU activity, with a baseline performance level and the ability to burst above that baseline.

In the Free tier template, only the Burstable classes are available.

Step 7: Select the storage type

Storage Types

There are 4 types of storage types available in the RDS storage configuration.

General Purpose SSD (gp2)

This storage type offers a baseline performance determined by the volume size. It's suitable for a wide range of workloads where the performance requirements are not consistently high.
General Purpose SSD (gp3)

With this storage type, performance scales independently from storage. It allows you to adjust performance and size separately, offering flexibility and cost-effectiveness for varying workloads.
Provisioned IOPS SSD (io1)

This storage type provides flexibility in provisioning I/O (input/output) operations per second (IOPS) to meet specific performance requirements. It's suitable for applications with demanding performance needs.
Magnetic

This storage type uses traditional magnetic hard drives and is limited to a maximum of 1,000 IOPS. It's generally not recommended for most modern workloads due to its relatively slower performance compared to SSD storage.

For this guide let us choose General Purpose SSD (gp2).

Storage Configuration

Allocated storage: This specifies the initial amount of storage allocated for your database instance, measured in gigabytes (GiB). The minimum value is 20 GiB, and the maximum value is 6,144 GiB.

Storage Autoscaling

Storage autoscaling: When enabled, this feature provides dynamic scaling support for your database's storage based on your application's needs. It allows the storage to increase automatically after the specified threshold is exceeded.
Maximum storage threshold: This specifies the maximum threshold for storage autoscaling.

Step 8: Setting up the connectivity

In connectivity of the RDS, there are two types of computer resources

Internal Access

Don’t connect to an EC2 compute resource

Select this option if you don't want to set up a connection to an EC2 compute resource for this database. You can manually configure the connection later if needed.

Connect to an EC2 compute resource

Choose this option to establish a connection to an EC2 compute resource for this database. This setting automatically adjusts connectivity settings to allow the compute resource to connect to the database.

Choose the appropriate option for your application.

Virtual Private Cloud

Select the Virtual private cloud (VPC)

Select the VPC where you want to deploy your database instance. The VPC defines the virtual networking environment for the DB instance. The default VPC typically includes subnets across multiple availability zones for high availability.

DB subnet group

Default: Choose the DB subnet group that defines the subnets and IP ranges the DB instance can use within the selected VPC. This ensures proper network isolation and security for the database instance.

Public access

Yes

If selected, RDS assigns a public IP address to the database, allowing Amazon EC2 instances and resources outside of the VPC to connect. Resources inside the VPC can also connect. You must choose one or more VPC security groups to specify which resources can access the database.
No

If selected, RDS doesn't assign a public IP address to the database. Only resources inside the VPC can't connect. You must still specify one or more VPC security groups to control access.

Since we are creating this RDS learning purpose choose option NO.

VPC security group (firewall)

Choose existing

Select existing VPC security groups that define rules for allowing access to your database. Ensure that the security group rules allow the appropriate incoming traffic.
Create new

If needed, create a new VPC security group to control access to your database.

Availability Zone

Choose an availability zone where the database will be deployed.

RDS Proxy

Create an RDS Proxy

This option allows you to create an RDS Proxy, which is a fully managed, highly available database proxy provided by RDS. RDS Proxy improves application scalability, resiliency, and security by efficiently managing database connections and pooling.

Certificate authority

Using a server certificate (optional): You have the option to use a server certificate for added security. This certificate validates that the connection is made to an Amazon database, providing an extra layer of security. The default certificate authority is "rds-ca-rsa2048-g1".

Step 9: Database Authentication

There are 3 types of authentication options available for AWS RDS

Password authentication

This option authenticates users using database passwords. Users provide their database username and password to access the database.
Password and IAM database authentication

In this option, authentication occurs using the database password and user credentials through AWS IAM (Identity and Access Management) users and roles.
Password and Kerberos authentication

This option allows users to authenticate with the DB instance using Kerberos Authentication, in addition to the database password. You can choose a directory in which authorized users can authenticate using Kerberos Authentication.

Password authentication will be selected as default.

Step 10: RDS Monitoring Setup

Turn on Performance Insights

Performance Insights provides a comprehensive view of your database's performance. Enabling it allows you to monitor performance metrics, query executions, and SQL statements in real-time.

The retention period for Performance Insights

This setting determines how long Performance Insights data is retained. In the free tier, Performance Insights data is retained for 7 days. After this period, the data is automatically purged. You can adjust this period based on your retention requirement.

Enhanced Monitoring

Enabling Enhanced Monitoring is useful for monitoring how different processes or threads utilize the CPU and identifying insights about RDS performance.

Step 11: Additional Configurations

Database Options

Initial database name: Specify the initial name for your database. If not specified, Amazon RDS will not create a database upon instance creation.
DB parameter group: Assigns a set of parameters to the database instance. The default parameter group for PostgreSQL version 16 is used.

Backup

Enable automated backups: Creates point-in-time snapshots of your database for data protection and recovery purposes.
Backup retention period: Specifies the number of days (1-35) for which automatic backups are kept. In this case, backups are retained for 1 day.
Backup window: Specifies the daily time range (in UTC) during which RDS takes automated backups. No specific window preference is set.
Copy tags to snapshots: Copies tags from the database instance to its snapshots for better organization and management.
Backup replication: Enables replication of backups in another AWS Region for disaster recovery purposes.

Encryption

Enable encryption: Activates encryption for the database instance. The default AWS Key Management Service (KMS) key "aws/rds" is used for encryption.

Log exports

Select the log types to publish to Amazon CloudWatch Logs: Specifies which types of logs (e.g., PostgreSQL log, upgrade log) should be published to CloudWatch Logs for monitoring and analysis.

Maintenance

Auto minor version upgrade: Enables automatic upgrades to new minor versions of the database engine as they are released.
Maintenance window: Specifies the period during which pending modifications or maintenance tasks are applied to the database instance. No specific window preference is set.

Deletion protection

Enable deletion protection: Prevents accidental deletion of the database instance by enabling deletion protection. While enabled, you cannot delete the database instance.

Step 12: Billing estimates

After configuring your RDS instance, review the billing estimates to understand the expected costs associated with your database deployment.
Take into account factors such as instance uptime, storage usage, data transfer, and any additional services enabled for monitoring or backup purposes.
Adjust configurations as needed to optimize costs and align with budgetary requirements.
Monitor your actual usage and costs regularly through the AWS Billing and Cost Management Console to track expenses and make informed decisions about resource management.

Step 13: After selecting all the options that satisfy the requirements of your application click on Create Database.

Step 14: Deleting your resources

When you no longer require your RDS instance or want to stop incurring charges delete the resources associated with the database.
Navigate to the RDS dashboard.
Locate the RDS instance you wish to delete and select it.
Click on the _Actions _ and choose Delete to initiate the deletion process.

Confirm the deletion by following the on-screen prompts and acknowledging any potential data loss.
If you enabled deletion protection, make sure to modify your database configuration before deleting the database. You can modify it by navigating to _Modify _ and disable the protection.
Review your billing statements to ensure that the resources have been successfully deleted and are no longer accruing charges.

RDS on AWS - Leave it or choose it !

EightPLabs — Wed, 06 Mar 2024 10:14:30 +0000

Introduction

A managed database service by Amazon Web Services (AWS).
They are designed to simplify database setup, operation, and scaling for relational databases.
RDS supports various database engines like MySQL, PostgreSQL, SQL Server, Oracle, and MariaDB.
It offers features like automated backups, monitoring, and high availability.
Ideal for businesses looking for reliable, scalable, and cost-effective database solutions without the challenge of infrastructure management.

Key Features of RDS

Availability: RDS offers automated backups and user-initiated snapshots for quick database recovery and monitoring. Snapshots can be shared among multiple AWS accounts for expanded availability while ensuring data security.
Security: Users create restricted passwords for database access and are assigned the Admin role by default. Encryption using KMS enhances data security.
Scalability: RDS allows for automatic scaling based on transaction volume, supporting both horizontal and vertical scaling to handle increased traffic or resource demands.
Performance: RDS offers SSD-backed storage options (General Purpose and Provisioned) impacting resource performance based on workload requirements.
Pricing: Pay-as-you-go model with no minimum charge, allowing users to delete resources and avoid charges. Free-tier accounts have specific configurations and no bills upon resource deletion.
Automated Backup and Redundancy: RDS provides automated backup mechanisms and redundancy features to ensure data integrity and availability reducing the risk of data loss.
Tools for Monitoring and Management: RDS offers tools for monitoring database performance, managing configurations, and diagnosing issues enabling users to maintain optimal database operation.
Automatic Software Updates and Other Security Features: RDS automatically applies software updates and patches to the database engine, enhancing security and compliance without requiring user intervention.

Available Database Engines

MySQL: Amazon RDS provides managed MySQL database instances, offering scalable, reliable, and cost-effective business solutions.
PostgreSQL: Amazon RDS supports PostgreSQL, allowing users to easily deploy and manage PostgreSQL databases, ensuring high availability and compatibility.
Oracle: Amazon RDS offers managed Oracle database instances, enabling businesses to run Oracle Database workloads in the cloud with features like automated backups and monitoring.
Microsoft SQL Server: Amazon RDS supports Microsoft SQL Server, providing managed instances for SQL Server databases with options for easy scaling and high availability.
MariaDB: Amazon RDS supports MariaDB, offering fully managed instances for MariaDB databases, allowing users to focus on application development while RDS handles database management tasks.

Available Server Types

Standard

Provides a balanced mix of computing, memory, and networking resources.
Suitable for a wide range of workloads with moderate resource requirements.
Offers predictable performance and cost-effective pricing.

Memory Optimized

Optimized for memory-intensive workloads that require high memory-to-CPU ratios.
Ideal for applications such as in-memory databases, caching, and analytics.
Provides ample memory capacity to handle large datasets and high-throughput workloads efficiently.

Burstable performance

Offers baseline CPU performance with the ability to burst CPU usage when needed.
Suitable for workloads with variable CPU demands or periodic spikes in activity.
Provides cost savings for workloads that do not require consistently high CPU utilization.

Optimized Read

It is designed for read-heavy database workloads that require high throughput and low latency for read operations.
Utilizes replicas with optimized configurations to offload read traffic from the primary database instance.
Improves overall database performance and scalability by distributing read requests across multiple replicas.

Should I go for it

YES!

Want to move faster
You lack manpower/time to manage DB
Money is not a constraint
Want to easily get many NFRs like backup/restore, redundancy , failover etc. with click of button

No!

Well, the biggest challenge is cost -if you are short on money, then RDS can start burning your reserve sooner than expected
You are techie, and can do a bit of hands dirty when needed
Still in POC mode and NFR like backup/restore, failover are not that crucial for you
Once DB is up, your interaction with it will be minimal (Minimal usage and less data so need of tuning etc.)

Wait for the next blog having detailed instructions as how to create a RDS DB in AWS

AWS S3 -solution architect certification guide

EightPLabs — Fri, 20 Jan 2023 11:32:14 +0000

What is S3?

S3 is an on-demand cloud storage solution managed by AWS that can be used to store files, data, images, videos, etc. Did you notice the “on-demand” word -it means there is no limitation on the size and as per demand size will be allocated. Also, since AWS manages it, users need not worry about scalability, availability, durability, and other non-functional aspects.

What are the uses of S3?

Backup and archiving
Storage of large files like video and images
Content sharing, S3 buckets can be shared across apps
Static website hosting
Bigdata processing

There are many other use cases that can leverage S3 but above are a few.

Advantages of using S3

Cost-effective: Storage in S3 is really cheap. There are some nuances with networking cost, still, it is much cheaper than other storage options available.
Auto-scaling: The user is concerned with adding data, there is no need for any capacity planning. This is a big plus. In the traditional disk-based approach, as soon as our data grows we need to scale the infrastructure which involves capacity planning, upfront expenses, and much more complexity.
High availability: S3 does not go down, literally, the data can be accessed from anywhere and anytime. If not using S3, programmers need to take care of this aspect, and can be a big distraction.
Durability: Data in s3 is stored at multiple places, so corruption at one place does not impact the data.
Global access: S3 buckets are global and can be accessed from anywhere in the world.

How S3 storage is different

Storage is of different types, for e.g., RAID, Object Storage -S3 falls into the category of Object storage.

Block storage: operates at raw storage device level and manages data as a set of numbered fixed-size blocks
File storage: Manages data as named hierarchy of files and folders.

Both of these are closely associated with the server and OS that is hosting the storage.

Object Storage: Contrary to these, S3 does not expose any underlying storage mechanism, it is managed as objects which can be accessed via API using standard HTTP verbs like GET, PUT, POST, etc.

Core concepts of S3

Buckets: Buckets are the primary containers to hold data. There is no limit on the number of objects which can be stored in the bucket. By default, the number of buckets in each account is 100 which can be increased to a max of 1000. There is no performance impact due to the number of buckets.
Objects: Objects are entities that we store in S3, e.g. files, photos, videos, etc. AWS treats objects as streams of bytes so there is no restriction as to what can be stored in the file. It can be as small as 0 bytes and as large as 5TB. Each object contains data as well as metadata. Metadata is a set of name-value pair which provides additional information about the object stored.
Keys: A key is a unique identifier for the stored object. They are used to access objects in the bucket and to perform other operations. It consists of the bucket name, path, and file name. If versioning is enabled, it is also part of the key. If there is a file named “test.jpg” in bucket “My_BUCKET” under path “test1/test2”
- The URL to access it will be https://MY_BUCKET.s3.us-west-1.amazonaws.com/test1/test2/test.jpg
- The bucket name is MY_BUCKET and the key is test1/test2/test.jpg
A common mistake, people take the “/” for folders but let’s repeat, there is no folder in S3, it’s just a convenient way to group stuff.

Accessing Objects

S3 is storage on the web, so every object has to go ta unique URL. From the above example, https://MY_BUCKET.s3.us-west-1.amazonaws.com/test1/test2/test.jpg represents a file that can be accessed from the web depending on the permission and policies.
S3 supports GET rest API as well as SDK through which Objects can be accessed
A unique link of an Object can be signed and made available as a public URL for a fixed period of time

Consistency

S3 provides two types of consistency

Read-after-write consistency: For creating and deleting existing objects, S3 provides read-after-write consistency -this means when a new object is created or existing once deleted, S3 guarantees that the latest version of the Object is available.
Eventual consistency: If objects are updated, it may take some time to propagate across all of its servers. It might be possible to read an older version of the object for a while even it got updated.

Static hosting

This is one of the prominent use cases, where static hosting can be enabled for a bucket and then the files inside the bucket can be used to host a static website. Typical steps would be:

Create a bucket and enable static website hosting for it
Add index and error files -these files are mandatory to be provided
Create a route53 record to point to the bucket
Create cloud-front distribution so that site is faster and more reliable

Versioning

Versioning can be enabled on S3 -by default, it is disabled. Once enabled, it can be suspended but cannot be disabled. When versioning is enabled, the files never get permanently deleted -older copies are maintained by AWS. This is very handy if we want to restore to an earlier version.

Access Control

Access policies can be used to define who can access what.

Resource-based policies
- ACL- Each ACL Identifies resources and the permission granted. The resource here can be a bucket or just an Object. It can be used to grant read-write permissions to other AWS accounts.
- Bucket policy- these can be applied at the bucket level and not the object level as in the case of ACL. Users or accounts can be granted access to content in a particular bucket.
User policies: AWS IAM can be used to create groups, and users and then assign them permission for specific buckets and specific operations on that.
Signed URLs: Presigned URLs can be created which will be public for a certain period of time and then will expire.
CORS: by default any client or website which is on a different domain, cannot access the resources of another domain. S3 allows enabling CORS for clients and domains so that its object can be cross-referenced.

Encryption

Securing data in transit
- S3 supports SSL/TLS to encrypt data in transit when it's sent to or retrieved from S3
Securing data at rest: Server side encryption with AWS managed keys (SSE-S3) is now applied by default on all the data stored in the S3 bucket. This change has been introduced on January 5, 2023, and this will not be charged separately.
- SSE-S3: This is the default and simplest approach. S3 automatically encrypts data when it's stored, and decrypts it when it's retrieved. The encryption keys are managed by S3 and are rotated regularly.
- SSE-KMS: Same as SSE-S3, but the Key Management Service (KMS) is used to manage the encryption keys. KMS provides a full audit trail of how the key has been used and modified over a period of time. This has additional cost implications.
- SSE-C: SSE-C allows you to provide your own encryption keys, which S3 will use to encrypt and decrypt your data. This option is helpful if you have existing encryption infrastructure or if you have specific compliance or regulatory requirements.
- Client-side encryption: Data uploaded to S3 is already encrypted.

Replication

Data in S3 is always replicated in multiple availability zones, but all zones are part of the same region. If for some reason, entire regions is unavailable, for e.g., natural disaster, political scenarios, etc. then we can lose data. Cross-region replication is a feature in S3 that can be used to replicate S3 objects in different regions. This comes at an additional price of storage, and network cost -so the pros and cost of cost vs the importance of data should be judged.

Event Notification

AWS S3 supports event notification which will notify registered listeners if some action happens on the object. For e.g, when an object is added/updated or deleted events will be generated which can be used to invoke lambda, send messages to SQS or SNS, integrate with event-bridge, and many more use cases.

What are the different storage classes in S3?

Depending on the access pattern of data we can store it in different classes of storage and further optimize on cost

Storage class for frequently accessed Objects: This storage class is most suitable for “hot” data, meaning this is the data on which an app or program operates frequently, is critical and needs a millisecond response.
- S3 standard: This is the default, if no storage class is identified, this will be used. Its availability is 99.99% (only this storage class has 2 9’s availability) and is available in >=3 Availability zones.
- Reduced Redundancy: This is intended for non-critical and reproducible data. But AWS does not recommend using it, S3 standard can have a better cost advantage.
Storage class for automatically optimizing data with changing or unknown access pattern: This storage class can be used for data that can be “hot” for a few days, but as it olds, it’s relevance may decrease.
- S3 intelligent tiering: This moves data to the most effective cost tier without impacting the performance impact. For e.g., if there is a news app that has videos and images, it makes sense to keep the latest data in S3 standard but move infrequently accessed data (as news grows older, the likelihood of it’s access also reduces) to cheaper storage. It’s availability is 99.9% and is available in >=3 Availability zones.
Storage classes for infrequently accessed objects: This is cheaper than the S3 standard but still provides millisecond access. This is most suited for data that is long-lived, is infrequently accessed, and still needs to be retrieved in milliseconds.
- S3 standard-IA: Data is stored in multiple availability zones and resilient to the loss of data in one availability zone. Its availability is 99.9% and is available in >=3 Availability zones.
- S3 one zone-IA: This is even cheaper than S3 standard-IA, but data will only be stored in a single availability zone. If that crashes, data will be lost. This is suitable for data for which replication is enables or data that can be easily recreated. Its availability is 99.5%.
Storage classes for archiving objects: When data is not frequently accessed and mostly will be needed on an “on-demand” basis, for example, to do some audit, or to analyze some data past. The most important aspect is, the time to retrieve data is not critical. Data in the glacier is first restored to S3 standard or S3 standard IA and then only it’s available for use.
- S3 Glacier instant retrieval: Although this data is archived, it can be accessed in milliseconds. Its availability is 99.99% and is available in >=3 Availability zones.
- S3 Glacier flexible retrieval: Effective when data is not accessed for at least 90 days. Retrieval can take minutes to hours. Its availability is 99.99% (after restore) and is available in >=3 Availability zones.
- S3 Glacier Deep Archive: This is the cheapest storage and should be used if data is not going to be needed for at least 180 days. Data can be retrieved before that but it may be subject to an early deletion fee. It has a default retrieval time of 12 hours. Its availability is 99.99% (after restore) and is available in >=3 Availability zones.

All storage classes except RRS (which is anyway not recommended by AWS) have durability of 99.999999999% (Nine 9’s). RRS (Reduced redundancy storage) has a durability of 99.99%

What are lifecycle rules?

Lifecycle rules can be used to transition objects from one storage class to another, delete, archive files or restore files from the glacier. For example, if we are using S3 to store application logs, we may need weekly logs for active analysis, but once they are older than 30 days we may want to archive them, and maybe after a year we no longer need them, so permanently delete the logs. These all transitions can be achieved automatically by setting transition rules.

Identity and Access Management In AWS

EightPLabs — Wed, 07 Dec 2022 10:55:44 +0000

This is one of the comprehensive and most pre-requisite topics on the path of learning AWS. If you refer to the documentation, it is whooping ~1200 pages! But do we need to know all? Well depends:

Are you going to be an AWS administrator who is going to manage company accounts, organization policies, etc. -then yes!
But for developers or solution architects who want to start building -only a subset of this is required.

The purpose of this series is to get started and get certified. We will start with the most basic concepts which will be sufficient for getting started and then in the next part, we will move on to more advanced concepts which will be helpful from point of view of certification.

What does IAM mean? IAM stands for Identity and Access Management :

Identity -who is trying to access. A user via console/CLI, or a program?
Access Management -well, managing which Identities(user/program) can access what?

Let’s not go into jargon -start analyzing, there are resources on amazon, e.g. DynamoDB, S3, Athena, SQS, etc. These all are shared and multiple users will be accessing it -but then how to restrict access, isolate usage of each user, and bill these users separately? These are the “resources” that need to be protected. Users or programs will have a different levels of access and usage:

Read-only
Read and write
Throughput requirements will vary for many users
Quotas will change for each user

So we have resources that need to be accessed in a restricted manner and in different ways -and we have users and programs who will be accessing these resources. How do we manage which users/programs have what permission on these resources? Policy or simply put it as a set of rules like:

Policy1
- Allow read-only access to S3
- Allow read-only access to DynamoDB
Policy2
- Allow read and write access to a particular bucket in S3
- Allow read and write access to all tables in Dynamo DB
- Allow deletion on S3

By default, everything is disallowed, so typically permissions need to be granted as what is allowed. What’s this now -permissions? The policy attached to a user or program defines what permission a user has on the resources. For example,

Policy1 above can be attached to regular users in an organization
Policy2 can be attached to admin users

So now consider this, if there are 10k users in an organization -are we going to create 10k Policies? Nope, here comes the group. We will group the users as per their role, like regular employees, sales, admin, etc, and then we will create a policy for each group.

In nutshell, AWS IAM tries to answer:

WHO → CAN ACCESS → WHAT?

Resource, user, and group are pretty straightforward, the crux of IAM is policies, how we define them, and then how we associate them with users, applications, or groups. Let’s dive a bit deeper into policies:

There are many more elaborate concepts but then I will not cover them here, I will have a separate write-up for that. Let’s do a quick example to understand what we have learned so far

Click on "create a new AWS account" if a first-time user or log in with your account if you already have signed up. Creating a new account is self-explanatory, it will need personal details, and card details so no point in showing it here. Everyone gets a free tier for 1 year, which will be sufficient for most of our experimentations. Also, AWS has quick labs which can be used to play around without paying any money.

Refer to video at end for more detailed steps and precise navigation

Log in to the AWS console
Select IAM from the services
Go to the "users" tab on the left side navigation panel
Click on Add users and provide user name
- Notice the "Aws credential type" section, there are two ways to access AWS services "Access key - Programmatic access" and "Password - AWS Management Console access"
- select "Programmatic access" as we intend to use this key to programmatically access the resources. The second option allows user to login in to the console -which we do not want at this stage.
For getting started easily, we will Attach policies directly. Click on tab "Attach existing policies directly". These are policies provided by AWS for most common permissions like read only for certail resources, complete access on resources etc. -we can select one of these. In future, we can also edit and further extend it. Start typing "AmazonS3" in "Filter policies" and all predefined policies related to S3 will appear. Let's select "AmazonS3ReadOnlyAccess".
Once a managed policy is attached, we are ready to create the user. Skip the tags for now.
After user is successfully created, it will navigate to screen having option to download creadentials for programmatic access of AWS resources in the boundary of permissions provided by policies. There are two parts
- access key and secret key
- Access key is visible by default but secret key is not. The keys should be downloaded and stored safely. Once we navigate away from the screen, there is no way to retrieve the secret key (Access key will always be visible).
- A new Access key and secret key can always be generated
But if we have thousands of users -we cannot keep attaching policy to each and every user. Here comes the concept of groups. We create groups and add users to the group. Attach policy to the group instead of attaching to individual user. Changing the policy at the group level will effectively change the policy for all of the users
- We can create a new group, for example, "S3_read_only". Select "User groups" from the left panel on IAM services
- If you notice, there is an option to add existing users to this group as well as assign a policy at the group level. As mentioned earlier, once a policy is assigned at the group level, it is applicable to all the users in the group.
- Next time, when we add a user, instead of attaching a policy, we can simply add them to a group and that policy will be applicable to all the users in the group.

Let's have quick look at the policy. Search for "IAM" service and select "User groups" from left navigation. Select the created group, for e.g., "s3_read_only_group" and go to the permissions tab. Click on the "+" on the policy as shown below

This will expand the policy as shown here

The policy is very easy to understand,

Line 5 indicates we are going to allow certain actions(line 7 to 10) on resources listed at line 12 to all the users who are part of this group
Line 7 to 10 identifies what actions are allowed -we can easily edit this and include or remove actions
Line 12 list the resources, '*" indicates all the buckets in S3, but we can easily restrict this to a particular bucket, or a bucket with certain name pattern

I will conclude the blog post with this summary

Resources -Services available on AWS which need to be secured
Policies -A set of rules governing what operations are allowed on each resource by a particular identity (User or Program)
Users -or rather more correct word is Principal -actual users or programs which will be accessing the resources as per rules defined in the policies
Groups -A set of users grouped together based on some role, for example, all developers, all testers, sales, marketing, etc.

Some highlights

IAM is universal → does not apply to Region. We will understand this more when we cover global infrastructure of AWS
When a new signs up for AWS, for e.g., we did during this tutorial, it is always root user. Do not forget to set up MFA on the root account. Multi factor authentication, single sign on, enterprise integrations -this will cover in next iteration of IAM
For new users, unless we attach explicit policy, NO permissions will be granted when they are created

We will need to learn at least one more service for example S3, then we can use these policies to access the resource and do some experimentation. I will be back soon with S3 and we will build our first application in the next session -so watch out.

Following is a video which explains the navigation in console in more details! Thanks and bye for now.

Study plan for certification, AWS Solution Architect -Associate

EightPLabs — Sun, 04 Dec 2022 05:47:59 +0000

In my last blog post, I emphasized the importance of getting certified especially “deep certified” and not “dump certified”. As promised, the following is a quick plan which we will follow to get to the goal of learning AWS, and as a by-product, we will also get certified.

Iteration 1, timeline ~(8 articles and 4 weeks), The purpose of this iteration is to get hands dirty rather than overload the mind with theoretical details. We will talk less about concepts and do more hands-on. Don’t worry, we will revisit the concepts later but this iteration will give you enough confidence to map your existing projects to a cloud one and you can claim to be cloud-ready! A quick outline:
- Understand the basics of AWS global infrastructure.
- Understand the basics of AWS security philosophy -this is the most fundamental aspect when building for the cloud as we operate in a shared model.
- Understand how to use groups, roles, and policies to authenticate and authorize users.
- Apply IAM policies, this is the bare minimum to get started developing solutions.
- Basic cloud formation script to create some users and resources via scripts.
- How to build APIs on amazon -APIs are lifelines, so we must understand them.
- Understand some basic services like AWS lambda, s3, dynamodb, Athena, etc. which will get us going and at the same time will not be very heavy.
- Build a real application -this is going to be fun, I will use an existing in-premise application and try to convert it to a cloud-based. Will share the code on GitHub.
Iteration 2, timeline ~(12 articles and 6 weeks), The purpose here will be to revise the last iteration and then build on some advanced concepts. By end of this iteration, you will have a very good hold on designing a scalable solution in AWS by using the appropriate choice of technology among many available.
- Revision with some sample papers and tests
- Extend security concepts and learn about single sign-on, network security, app security, token-based authentication, etc.
- Learn more about building scalable architecture like on-demand scaling, managing a growing database, routing, content delivery, and many more.
- API management, horizontal scaling vs vertical scaling, event-driven architectures, serverless architectures, queue and messaging concepts -these all will help you build real-time scalable system design rather than just mugging up hypothetical system design courses
- How to make choices for different options like relational vs NoSQL or dynamo vs Athena -when to use what
Iteration 3, timeline ~(4 articles and 2 weeks), This will be the last iteration on our learning path for the associate exam. We will cover some advanced theoretical concepts, no need to get hands dirty there -at least from a certification point of view. We will cover
- Start with revision papers and some summary from the last two iteration
- Data analytics solutions and concepts
- Geo redundancy, backup recovery, and other non-functional concepts
- A bit more internals of networking like Internet gateways, route tables, etc.
- Cost optimizations, AWS cost management tools
Iteration 4, timeline ~(10 revision papers and 1 week) -7 days and 10 papers, this will be the most comprehensive week from the perspective of exam preparations. We will concentrate more and more on practice.

The ultimate goal of the Amazon solution architect exam is to crack this ratio.

And we surely will but instead of going waterfall which is so old fashioned ;), we will hit the buzzword -agile! We will take a slice from all 4 domains in each of the iterations, build something, validate our previous knowledge and then keep advancing. At end of each iteration, we will be more and more cloud-ready and also a lot near our goal post of getting certified.

I will publish these tutorials on youtube also, many are comfortable in more interactive mode rather than just text! This is the introductory video

Watch out for the next blog and video covering IAM -no more plans and intros, let's get our hands dirty!

Are certifications money wasted or invested?

EightPLabs — Wed, 23 Nov 2022 03:19:56 +0000

There have been a lot of debates around whether one should "invest" in certifications or just be away from "wasting" money there. I have been in the same boat and I do have at least 5 certifications (Java language, Java architect, AWSx2, Spring), so I feel I am qualified to share my view on this topic.

For people in a hurry, YES, it is never "money wasted", it is always "the money invested". One prominent argument against certification is that most people use dumps to get certified and certificates do not exhibit real knowledge -there is truth in there, still, I would advocate getting one. Let me list the benefits of being certified

Resume selection -this itself is a big advantage, many times we all struggle to get that first call! In today's era of bots, what keywords are there on the resume matters a lot! They will simply increase the ranking for certified professionals -bots cannot differentiate whether it's from the dump or deep knowledge. Even when a resume goes to a recruiter, they tend to look at buzzwords, the way we try to judge a doctor's knowledge by the number of diplomas on board! Just human nature! The call itself is a big headstart and it can be used iteratively to build on expectations. No exception, what follows is based on our knowledge but at least we get going!
One learns in the process: Yes, if you just used dumps, you may not be able to justify the knowledge but it has given a headstart. Keep building based on interviews -understand the gap, and learn once onboarded.
You are not lost in the ocean: Have you tried looking at all the services offered by AWS, got in the feeling of where to start and how to start? We literally get lost -certification provides a stepwise framework.
Connect the dots: This is a very important point. Certification provides a breadth, we know our options. We build on the minimal knowledge and then we can choose how deeper we want to go.
Helps with side gigs: Many organizations, especially startups cannot afford full-time employees. Having a certificate helps in landing that first gig, after that it's all up to the person if he got "dump" or "deep" knowledge. It can also help in teaching gigs, with the market becoming competitive day by day, training and certification will definitely be a big area.

I rest my case -Let's keep it concise.

I am a certified AWS professional solution architect. I am going to start a series of blogs targeting AWS solution architect -Associate and then will move on to AWS solution architect -Professional. No this is not going to be #AWeek or #30Days, we will go deep starting right from where to begin, what is the minimal knowledge needed for being called a cloud professional. Here is a bit of an outline of how we will progress:

Publish Plan
Publish gradual reading material
Weekly live sessions on zoom
Career Guidance and resume consultation
Practice papers

So what are you waiting for? Do connect me on LinkedIn, this will be fun and it's all free -so even more fun!