Forem: Payal Gupta

Glue cross-account setup

Payal Gupta — Sat, 11 Jan 2025 16:10:44 +0000

This document will cover detailed steps on how to query glue DB catalog from Dremio in a cross-account setup using AWS Lake formation

Use-case
Account A - Dremio is deployed here and AWS Glue_DB_A is created and added as a source in Dremio

Account B - AWS Glue_DB_B is created and data is located in the S3 bucket

Customer wants to share Glue-DB B catalog with Glue-DB A and query the data located in account B from Dremio

Setup Diagram

Role of each of service in the given setup -

Lake Formation - To create data mesh, simplify cross-account data sharing, and create resource links
Resource Access Manager - To share resources and view shared Data catalog
IAM User - To provide cross-account read/write access to the S3 bucket to run queries from Dremio
Amazon Athena - Just to test whether lake formation access is working fine or not

Steps

Resource Sharing using Lake Formation and Resource Access Manager

First we need to use Lake Formation and Resource Access Manager to share glue catalog from account B to A

Steps for Account-B:

Create Glue DB named Glue_DB_B
Create Glue Table in this DB, point to S3 location where data resides, and provide schema
OR
You can use glue crawler to automatically extract data from S3 and add glue table for you.
Go to Lake Formation console -> Data Lake Location -> Register same S3 location -> Use default IAM role -> AWSServiceRoleForLakeFormationDataAccess
Go to Lake Formation -> Databases -> Select Glue_DB_B -> Actions -> Grant -> Fill in (External Account), put AWS Account-A ID -> Choose a specific table

For DB, grant Alter, Create table, Describe
For Table, grant Alter, Delete, Describe, Drop, Insert

Go to Resource Access Manager console -> Shared by me in the left pane -> Resource Shares You should be able to view your shared resources

Steps for Account-A:

Go to Resource Access Manager → Shared with me → Resource Shares → Accept your Resource Share
Now, Go to Lake Formation -> Table -> Your shared table will appear here -> Click on table -> Actions -> create Resource link
Table will now appear italicized in the glue db as shown below

Provide cross-account read/write access to the S3 bucket

Steps to do so:

Go to Account B → S3 console
Select your S3 bucket
Go to the Permissions tab
Edit Bucket Policy and add the following policy (make sure to add the AWS Account-A ID, IAM User name, and bucket name)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AccountA-ID>:user/<username>"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket-name>/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AccountA-ID>:user/<username>"
            },
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<bucket-name>"
        }
    ]
}

Add Glue catalog as a source in Dremio

Last step is to add Glue_DB_A as a source in Dremio :

Go to Add Source
Select AWS Glue Data Catalog
Fill in the details - Name, Region, Authentication
Hit Save

You should be able to view the datasets from both the glue catalogs and run queries on them.

You can run the query on the glue source via Athena instead of Dremio.

How to use Glue crawler to add tables automatically

Payal Gupta — Sat, 07 Dec 2024 12:54:12 +0000

This document will cover the steps on how to use Glue crawler to extract data from S3 to automatically add tables to the glue DB and run queries on it from Dremio or Athena

Setup Diagram

Steps to follow

Create an S3 bucket and upload the raw data i.e, csv, json files.
Go to AWS Glue Console and Create Glue DB
Go to Tables page and Select Add Tables using crawler on the top right corner

This should land you to the AWS Glue Crawler setup page

Follow below steps to fill in the details

Name - Enter the Crawler name
Add data source
- Data source - Select S3
- Location of S3 data - Select In this account (if that’s the case)
- S3 path - Browse for the S3 bucket which contains the data and don’t forget to add forward slash at the end
- Subsequent crawler runs - Select Crawl all sub-folders
Click Add an S3 data source
Click Next → Configure security settings
Click Create new IAM role and give a name to the role. It will create a new IAM role required by the Glue crawler to extract the data present in the S3 bucket
Next, Set output and scheduling
- Select the Target Database - you can choose default or create a new one
- Crawler schedule - On Demand
Next → Review and Create → Create Crawler
Now, the crawler has been successfully created and you can run the crawler

It will take few minutes to extract the data from S3 bucket and once it is done, you should see the state as Ready

Now, you should be able to see a table added in the glue DB

Go to Dremio → Add the glue catalog as a source
Name - Enter glue catalog name
Region - Select the AWS region
Authentication - AWS Access key

Click Save and run queries on the glue DB from Dremio! or Athena

Embracing the Future of Tech: Attend AWS re:Invent 2023 as an All Builders Welcome Grant Recipient

Payal Gupta — Wed, 25 Oct 2023 12:43:26 +0000

The journey of a thousand miles begins with a single step, they say. For me, that first step was the moment I received an email that would forever change my perspective on the future of cloud computing and digital transformation.

Last year, I received the All Builders Welcome grant to attend AWS re:Invent 2022!

But unfortunately, due to the delay in getting US visa, had to defer it to this year.

I am thrilled to share that I got the US visa this year and will be attending AWS re:Invent 2023 in Las Vegas as an ABW Grant recipient!

The All Builders Welcome Grant: A Gateway to Knowledge and Opportunity

AWS re:Invent is renowned as the largest gathering of the global cloud computing community. Each year, it attracts an array of visionaries, engineers, and experts who are at the forefront of technological advancement.

The grant includes registration for re:Invent 2023, airfare to Las Vegas, Nevada, and hotel accommodations for the duration of the conference.

Participants will have access to five days of re:Invent content and activities as well as a curated program designed to remove barriers and create opportunities for learning, career growth, and community building, including a welcome event, reserved seating in the keynotes, meetups, a mentoring luncheon with AWS leadership, and much more!

To learn more about it, check out the official page - https://reinvent.awsevents.com/community/all-builders-welcome/

The Significance of AWS re:Invent

AWS re:Invent is more than just a conference; it is a hub of innovation, knowledge-sharing, and networking. It is where the brightest minds come together to discuss, dissect, and debate the future of cloud technology and its applications. The event covers a vast spectrum of topics, from machine learning and artificial intelligence to serverless computing and security.

What to Expect

As I prepare to embark on this incredible journey, I'm excited about several aspects of the event:

Knowledge and Insights:
AWS re:Invent is an unparalleled source of knowledge and insights. It's a place where you can learn from the best in the industry, stay updated on the latest developments, and gain a deeper understanding of cloud technology's future.
Networking:
The event provides a unique opportunity to connect with professionals, experts, and peers who share my passion for technology. These connections can lead to collaborations, idea exchanges, and lifelong friendships.
Innovation:
Expectations for groundbreaking announcements and product launches run high at AWS re:Invent. It's the place where the technology of tomorrow is unveiled today.
Personal Growth:
Beyond the technical aspects, attending a conference of this magnitude can help me grow as an individual and professional. It will broaden my horizons and challenge my thinking.

Thank You, AWS

I want to express my profound gratitude to the AWS team for providing this incredible opportunity to me. It is not only an acknowledgment of my passion and dedication but also an investment in my future. I look forward to representing the All Builders Welcome community at AWS re:Invent and making the most of this experience.

Let's Connect!

As I prepare to embark on this incredible journey, I want to invite you all to join me. If you're attending AWS re:Invent 2023, I would be delighted to connect with you. Let's learn, share, and innovate together.

Stay tuned for live updates from the event, where I will be sharing key takeaways, exciting announcements, and my personal reflections on this journey of knowledge and discovery.

The future of technology is unfolding before us, and I can't wait to be a part of it at AWS re:Invent 2023!

AWS Solutions Architect Professional Exam Preparation Guide

Payal Gupta — Sat, 10 Dec 2022 07:14:04 +0000

This blog post will cover:

Brief overview of AWS Solutions Architect Professional Exam
Who should take this exam?
Some useful tips & Resources to crack the exam (based on experience)

Let's get started...

Overview

This is a professional level exam and it is considered as the toughest of all the AWS Certifications. It needs extensive knowledge and abilities in offering complicated solutions to complex problems in order to maximize security, cost, and performance, and automate manual procedures.

Exam will include 75 questions, either multiple choice or multiple response to be completed in 180 minutes.

Who Should take this exam?

This exam is intended for individuals with two or more years of hands-on experience on AWS to design and deploy cloud architecture to provide better solutions to complex problems.

You would need a deeper understanding of the AWS services in terms of how these services work together along with their best practices and useful benefits over one another in order to choose the best option for the given use case.

Some useful tips & resources to crack this exam

I recently took this exam and was able to crack it by following below practices. Hope they help you too.

Do hands-on labs
Take any good Course available on Udemy, Whizlabs, CloudAcademy etc.
I took Cloud Academy course that helped me alot.
Mock test series
Make sure to atleast take 2 mock tests in a week for practice. I used Mock test series by Stephane on Udemy.
Read AWS Services FAQs and Whitepapers
Many times the questions come from the FAQs and whitepapers as these resources includes how to best implement the AWS services, and how one service is different from the other one in a particular use-case. So, I would highly suggest you to go through them at-least once.
Learn about AWS Services best practices
Practice Time and Focus Management
This is crucial to crack the exam. One thing that absolutely helped me in cracking this exam was the daily routine that I created for this exam's preparation.
You need to ensure that you block your calendar or put a reminder to dedicate some focused time only for exam preparation such as for doing mock tests, reading through whitepapers etc. To know more, check out this video.

That's all for now. Thanks!

AWS Solutions Architect Associate Exam Preparation Guide

Payal Gupta — Sat, 01 Oct 2022 12:45:48 +0000

This blog will cover some useful resources and the exam preparation plan to help you pass the AWS SAA certification.

First thing to mention, do not consider this exam to be easy :)

AWS SAA is one of the popular and most challenging exams in the cloud domain which helps you gain knowledge and boost your confidence in using AWS cloud services. So, it’s great that you are planning to take this exam and up-skilling your cloud knowledge.

Now, let’s talk about the resources to prepare for it. As mentioned, this exam is not an easy one, it will test how good you are with AWS services in terms of their usage, best practices, designing and implementing distributed systems on AWS. So, below are some of the good resources which I found helpful.

Courses:

Udemy: Ultimate AWS Certified Solutions Architect Associate (SAA) by Stephane Maarek

Udemy: AWS Certified Solutions Architect Associate Training SAA-C03 by Neal Davis

Cloud Academy: https://cloudacademy.com/learning-paths/aws-solutions-architect-associate-saa-c02-certification-preparation-954/

Practice Tests:

Whizlabs: https://www.whizlabs.com/aws-solutions-architect-associate/

Udemy: https://www.udemy.com/course/aws-certified-solutions-architect-associate-amazon-practice-exams-saa-c03/ by Jon Bonzo

Sample Example Questions by AWS: here

AWS FAQs: here

AWS Whitepapers & Guides: here
Some of the important white papers are –

Overview of Amazon Web Services
AWS Well-Architected Framework
Architecting for the Cloud: AWS Best Practices
AWS Security Best Practices
AWS Storage Services Overview

AWS Ramp-Up Guide: Architect: here

Learning Path

On average, 35-40 hours or 6-8 weeks should be fine to prepare for AWS SAA exam (if you have some prior AWS experience).

If you are just beginning with AWS, then you might need approximately 50-60 hours or three months to prepare.

Note: These recommendations are in general, and not a hard constraint. Please feel free to go on your own pace which suits you better.

Below is the 3-months learning path for beginners:

Month 1: Get familiar with AWS Services and their best practices

Course/youtube Videos - 10 hours
Practice labs/hands-on - 10 hours

Month 2: Deep Dive! Learn about different use-cases and how to implement them using AWS Services such as which service would be the best fit in the given particular scenario and why

Course/youtube videos/AWS Ramp-Up Guide: Architect - 10 hours
Read AWS Whitepapers/FAQs - 5 hours
Hands-on - 5 hours

Month 3: Practice Mock tests!! To pass the exam, practice questions are extremely important. Take any one of the above mentioned practice test series, try to go through all the test questions, check the detailed explanation provided to understand why that particular option was the right one to clear your concepts which will help you perform better in the exam.

Hope it helps. Good Luck!!

Let's start with AWS Lambda

Payal Gupta — Tue, 15 Feb 2022 18:28:40 +0000

This post will cover the following content:

What is AWS Lambda?
Why should you use AWS Lambda?
How to use AWS Lambda?

Let's get started...

What is AWS Lambda?

Lambda is a highly available, serverless, event-driven compute service that lets you run code without provisioning or managing servers or clusters. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.

Why should you use AWS Lambda?

Lambda is best suited for shorter, event-driven workloads, since Lambda functions run for up to 15 minutes per invocation.

Also, when you use Lambda, you are only responsible for your code and Lambda will take care of the rest i.e, balance of memory, CPU, network, and other resources to run your code.

This would mean that you cannot log in to compute instances or customize the operating system on provided runtimes because Lambda will perform operational and administrative activities on your behalf, including managing capacity, monitoring, and logging your Lambda functions.

If you are looking to manage your own compute resources, you can use EC2 or EBS (Elastic Beanstalk) as per your requirements.

How to use AWS Lambda?

You can create, invoke, and manage your Lambda functions using any of the following interfaces:

AWS Management Console – Provides a web interface for you to access your functions.
AWS Command Line Interface (AWS CLI) – Provides commands for a broad set of AWS services, including Lambda, and is supported on Windows, macOS, and Linux.
AWS SDKs – Provide language-specific APIs and manage many of the connection details, such as signature calculation, request retry handling, and error handling.
AWS CloudFormation – Enables you to create templates that define your Lambda applications.
AWS Serverless Application Model (AWS SAM) – Provides templates and a CLI to configure and manage AWS serverless applications.

For demo purpose, I will be using AWS Management Console to get started with Lambda service.

So, let's create our first Lambda function using console...

Go to Lambda Service in the console
Go to Functions page
Click Create Function. You should see the below flyout.

Now, in this flyout, there are various option that we should understand. Let me help you with that.

First, you need to choose one of the options out of the following to create a function

Author from scratch: This is the default code that Lambda creates to start with a simple "Hello World" example.

Use a blueprint: This option is to create a Lambda application with sample code and setup configuration for typical scenarios.

Container image: Choose this option if you have a container image which you would like to use to deploy a Lambda function

Browse serverless app repository: Choose this option for deploying serverless applications from the AWS Serverless Application Repository

For now, I will go with the first option "Author from scratch" and fill the basic information:

Function name: Enter a name that describes the purpose of your function. I've entered "Lambda-demo"

Runtime: Choose the language to use to write your function. Note that Lambda provides runtimes for .NET (PowerShell, C#), Go, Java, Node.js, Python, and Ruby. I'm selected "Node.js 14.x"

Architecture: Choose the instruction set architecture you want for your function code. Keeping default i.e, x86_64

Permissions: By default, Lambda will create an execution role with permissions to upload logs to Amazon CloudWatch Logs. You can customize this default role later when adding triggers.

You have three options to choose for Execution role.

Create a new role with basic Lambda permissions - Default option
Use an existing role - if you have an existing IAM role to use, you can choose this option
Create a new role from AWS policy templates - use this option if you would like to use AWS provided policy template for various services

I am going with the default option i.e, "Create a new role with basic Lambda permissions"

After filling all the above details, Choose "Create function" in the bottom right corner.

Now, let's invoke the Lambda function which you created...

Steps to Invoke Lambda Function:

Under Functions, select your lambda function which you want to test.
Go to Actions on the top right corner -> select 'Test' from the drop down.
You can invoke your lambda function with a test event.You can either choose a template that matches the service that triggers your function, or enter your event document in JSON.

I will be using 'hello world' event template provided by AWS.

So, choose 'New event'
For template, select 'Hello World' from the drop down
Template should look like this:

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

Provide a name to your event function e.g. 'demo-event'

Now, click 'Save changes' and choose 'Test'

You should see the Execution results as below:

We can see the 200 OK response, memory used, billing duration for lambda function, total duration, etc in the summary section.

Further, under Log Output, you can see the logs generated for the test event or you can click on the 'Click here' which will direct you to view the same logs under the corresponding CloudWatch log group.

Lambda Monitoring

You can view the monitoring details of a Lambda function under the 'Monitor' tab.

Lambda sends runtime metrics for your functions to Amazon CloudWatch. Logs all requests handled by your function and automatically stores logs generated by your code through Amazon CloudWatch Logs.

As you can see in the screenshots, you can also view these logs under CloudWatch console by clicking 'View logs in CloudWatch' option.

Execution Role

In the end, you can check the details for the Execution Role that grants the function permission to access AWS services and resources. In our demo, this role was created when we created the Lambda function above.

In order to check execution role details of your Lambda function, go to 'Configuration' tab -> then select "Permissions' from the left pane. You should see the details as follows:

View the resources and actions that your function has permission to access by choosing 'By action' or 'By resource' column.

That's all I wanted to cover in this blog. Hope it was helpful. Thank you.

Working with Placement groups in Amazon EC2

Payal Gupta — Sat, 05 Feb 2022 20:12:16 +0000

This blog will cover the following content:

What is a Placement Group in Amazon EC2?
What are the benefits of using placement group?
How to create placement group, and launch instances in placement group?

Let's get started...

What is a Placement Group in Amazon EC2?

Placement group is a way to impact the placement of interdependent EC2 instance groups in order to suit your workload requirements. AWS provides three placement strategies which you can use based on the type of your workload:

Cluster Placement Groups: A logical grouping of instances within a single AZ.
Partition Placement Groups: Logical partition of instance groups such that no two partitions within a placement group share the same underlying hardware.
Spread Placement Groups: each instance within a spread placement group will be placed in a different rack.

What are the benefits of using placement group?

Cluster Placement group benefits:

Recommended for low network latency, and/or high network throughput applications.
Only specific to a single AZ
Can span peered VPCs in the same Region

Partition Placement Group benefits:

Reduces the impact of correlated hardware failures for your application
Mainly used to deploy large distributed and replicated workloads, such as HDFS, HBase, and Cassandra, across distinct racks.
Can have partitions in multiple Availability Zones in the same Region.
Offer visibility into the partitions using which you can check which instance is in which partition. Topology-aware applications, such as HDFS, HBase, and Cassandra use this information to make intelligent data replication decisions for increasing data availability and durability.

Spread Placement Group benefits:

Recommended for applications that have a small number of critical instances that should be kept separate from each other.
Reduces the risk of simultaneous failures that might occur when instances share the same racks which is not the case in spread placement group
Can span multiple Availability Zones in the same Region.

How to create placement group, and launch instances in placement group?

Before using placement groups, I would suggest you to go through the rules and limitations of placement groups once for awareness.

To create a placement group,

Go to Amazon EC2 console
In the left pane, go to Network & Security -> choose Placement Groups -> Create placement group.

Fill the details in the fly-out
Name: Specify the name of your placement group
Placement strategy: choose the strategy from the drop-down
Tags: Optionally assign tag values to the placement group
Click Create Group

In my account, I created 3 placement groups with cluster, spread, and partition startegies as shown below.

Launch instance in placement group

Go to EC2 console -> Instances
Click launch instances in the top right corner
Launch instance with the following steps:
Step 1: Choose AMI as per your requirements
Step 2: Choose instance type by keeping in mind the limitations of your placement group. Example: You cannot lanuch t2 type instances in Cluster placement group because burstable performance instances such as T2 are not supported by cluster placement group. Hence, make sure to choose the instance type which is supported by the placement group in which you are planning to launch it otherwise, you will receive an error message as shown in the screenshot below.

Step 3: Configure Instance Details This is the step where you will specify the instance details required for your placement group.

Number of instances: Enter the total number of instances that you need in this placement group, because you might not be able to add instances to the placement group later.

Placement group: Select the Add instance to placement group check box

Placement group name: You can choose to add the instances to an existing placement group or to a new placement group that you create

Placement group strategy: Choose the appropriate strategy.
If you choose partition, for Target partition, choose Auto distribution to have Amazon EC2 do a best effort to distribute the instances evenly across all the partitions in the group. Alternatively, you have the option and control to specify the partition in which to launch the instances.

Step 4: Add Storage to your instances
Step 5: Add tags such as Name tag
Step 6: Configure Security group for your instances
Step 7: Review and Launch
If everything looks fine, click launch in the bottom right corner to launch the instances.

I created 3 instances in partition placement group as shown below.

Additionally, to check the placement group details of your instance, select the instance -> go to Details section of your instance and scroll down to Host & placement group section. You can find the placement group name, and partition number etc.

Hope this information is helpful. Thank you.

Connect with Couchbase Capella over Private network created using AWS VPC Peering

Payal Gupta — Sun, 23 Jan 2022 10:10:33 +0000

This blog will cover the following things:

What is Private Network in Couchbase Capella?
What is AWS VPC Peering and how Private network is created using it?
How to connect with Capella over private network?

Let's get started...

What is Private Network in Couchbase Capella?

Capella is a fully managed Database as a Service (DBaaS) offered by Couchbase which provides easiest and fastest way to begin with Couchbase database and eliminate your database management efforts. You can easily deploy a clustered database in the public cloud such as AWS/Azure/GCP using Couchbase Capella.

Private network is the feature provided by Capella using which you can connect your application with the Couchbase Capella Cluster over a private connection. It enables you to have more secure connection with less latency and data egress costs.

What is AWS VPC Peering and how Private network is created using it?

AWS VPC Peering is the private network connection created between two VPCs that can be in the same or another AWS account, to route traffic between them using private IPv4 addresses or IPv6 addresses and access or share the resources created in those VPCs as if all the resources are a part of the same network.

Couchbase Capella uses AWS VPC Peering to create a Private Network connection between your application and the Capella Cluster.

How to connect with Capella over private network?

Let me help you with the step-by-step guidelines to create a Private Network connection in Capella below:

Note: Before beginning, please make sure of the following things:

Route53 should be enabled on your AWS account

Your application VPC and Couchbase Capella cluster VPC have different CIDR. If both VPCs will have overlapping CIDR blocks, then we won't be able to setup peering connection due to the limitation of VPC peering.

So, let's get started with the setup...

Login to Couchbase Capella console and create a cluster using Couchbase’s Cloud Account option for free.
You can create the cluster using your own Cloud account option as well. However, I am sharing the steps using Couchbase Cloud Account option so would be easier for you to follow.
Created private network between Capella Cluster VPC and your own VPC in which your application resides. Steps are as follows:

Go to Clusters in the left pane
Go to Connect tab > Virtual Network > Manage Private Network
On the top-right corner, click on Setup Private Network
Confirm the pre-requisites Route53 Enabled Virtual Network Peering Enabled
Enter the following details
Name - Your Private Network name which will be visible on the Capella UI
AWS Account ID - Your AWS Account ID in which your application VPC resides
Virtual Network ID - Your application VPC ID
Available Regions - AWS Region in which your application VPC exists
CIDR Block - CIDR block of your app VPC
Now, run the commands shown on your Capella UI

aws ec2 accept-vpc-peering-connection --region=<> --vpc-peering-connection-id=<pcx-xxxxxxxxxx> -> for accepting vpc-peering connection (this can be done via AWS console also)

aws route53 associate-vpc-with-hosted-zone --hosted-zone-id=<> --vpc=VPCId=<>,VPCRegion=<> --region=<> -> for associating VPC with the hosted zone in route53

By now, your peering connection between Capella VPC and application VPC has been created. Next step would be to add the routes in your app VPC's route table in order to communicate.
Capella Cluster VPC CIDR can be found in the AWS VPC peering console
-> Login to your AWS account in which app VPC resides
-> Go to VPC peering console
-> Search for the peering connection using the ID provided in above commands
-> You should see the requester VPC details, copy the Requester CIDRs
-> Now, Go to the your app VPC route table
-> Go to Routes tab > Click on edit route
-> Add Capella VPC CIDR in destination and select peering connection as target
Private network setup is completed at this point. Now, it's time to test the connection if it is working fine or not.
You can use the below commands to test the private connectivity with Capella Cluster.

nslookup output for DNS resolution:

[ec2-user@ip-192-0-0-4 ~]$ nslookup -type=SRV _couchbases._tcp.cb.uvbaw6f5kvhmun7s.cloud.couchbase.com
Server:  192.0.0.2
Address:    192.0.0.2#53

Non-authoritative answer:
_couchbases._tcp.cb.uvbaw6f5kvhmun7s.cloud.couchbase.com    service = 0 0 11207 yk9iixsbth4mj5uf.uvbaw6f5kvhmun7s.cloud.couchbase.com.
_couchbases._tcp.cb.uvbaw6f5kvhmun7s.cloud.couchbase.com    service = 0 0 11207 20pcyksdifyr2r2s.uvbaw6f5kvhmun7s.cloud.couchbase.com.
_couchbases._tcp.cb.uvbaw6f5kvhmun7s.cloud.couchbase.com    service = 0 0 11207 cxq21w9wmkbl90em.uvbaw6f5kvhmun7s.cloud.couchbase.com.

Netcat or telnet command to test connectivity

telnet output:

[ec2-user@ip-192-0-0-4 ~]$ telnet yk9iixsbth4mj5uf.uvbaw6f5kvhmun7s.cloud.couchbase.com. 18091
Trying 10.0.113.52...
Connected to yk9iixsbth4mj5uf.uvbaw6f5kvhmun7s.cloud.couchbase.com..
Escape character is '^]'.

netcat output:

[ec2-user@ip-192-0-0-4 ~]$ nc -v 20pcyksdifyr2r2s.uvbaw6f5kvhmun7s.cloud.couchbase.com. 11207
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.112.74:11207.
^C
[ec2-user@ip-192-0-0-4 ~]$ nc -v yk9iixsbth4mj5uf.uvbaw6f5kvhmun7s.cloud.couchbase.com. 11207
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.113.52:11207.

Traceroute output to confirm the path taken:

[ec2-user@ip-192-0-0-4 ~]$ sudo traceroute 20pcyksdifyr2r2s.uvbaw6f5kvhmun7s.cloud.couchbase.com. -T -p 18091
traceroute to 20pcyksdifyr2r2s.uvbaw6f5kvhmun7s.cloud.couchbase.com. (10.0.112.74), 30 hops max, 60 byte packets
 1  ip-10-0-112-74.ec2.internal (10.0.112.74)  1.292 ms  1.279 ms  1.271 ms

Hope this information was helpful. Thank you.

Create an Organization in AWS

Payal Gupta — Sat, 08 Jan 2022 13:37:51 +0000

This post will cover the following content:

What is AWS Organizations?
Why should you use AWS Organizations?
How to use AWS Organizations?

Let's get started...

What is AWS Organizations?

AWS Organizations is a service that comes under AWS M&G category, which helps you centrally manage multiple AWS Accounts and govern your environment as you grow and scale your AWS resources.

Why should you use it?

You should use AWS Organizations if you would like to

Centrally manage your environment across multiple AWS accounts
With the help of AWS Organizations, you can

Create new accounts that will automatically be a part of your Organization
Invite other AWS Accounts to join your Organization
Programmatically create new AWS accounts to quickly scale your workloads
Attach appropriate policies to apply on some or all of the accounts.
Use consolidated billing feature to consolidate and pay for all member accounts which helps to manage billing and cost centrally.

Manage your Organization

Group AWS accounts into Organization Units(OUs) to easily manage and govern the boundaries based on service control policies(SCPs) for your OUs.

Simplify permission management and access control

Using AWS SSO(Single-Sign-On) and Active Directory, control user-based permissions in your Organization.
Apply SCPs to control access to AWS Services within OUs.

Efficiently share and provision resources across accounts

Use AWS RAM(Resource Access Manager) to share critical resources within your Organisation to help reduce resource duplication.
Use AWS License Manager to centrally meet your software license agreements
Use AWS Service Catalog to easily share a catalog of IT services and custom products across accounts

Manage costs and optimize usage

AWS Organization provides the feature of Consolidated billing to have shared billing functionality which enables the management account of your organization to pay for all the member accounts and take benefit of quantity discounts with a single bill.
Use AWS Cost Explorer to track resource costs
Use AWS Compute Optimizer to compute resource usage

Audit your environment for compliance

You can use various AWS services with Organisations to centrally manage security of your resources such as
AWS CloudTrail to audit all the events in your accounts
AWS Config to centrally define your recommended configuration criteria across resources, AWS Regions, and accounts
AWS GuardDuty for threat detection to protect your resources centrally
AWS Control Tower to establish cross-account security audits, or manage and view policies applied across accounts

How to use AWS Organizations?

AWS Organizations is a global service and you can use AWS console, CLI, or API to create and use AWS Organization service.

I will be performing the following steps to show how to use this service:

Creating an Organization
Invite and Add other AWS account to my Organization
Create Groups (OUs) within my Organization
Apply SCPs(Service Control Policies) to the Group

Note:
I already have two separate AWS accounts. Will be using one as Management Account (formerly known as master account) and other as Member Account. Management account is the account that is used to create Organization and Member accounts are all the other accounts which you invite or create within an Organization.

Steps are as follows:

Login to your Management Account and go to the AWS Organizations console
Click "Create Organization" in the top right corner
You should receive a verification email sent to the email address associated with your Management Account in order to verify the account. You need to verify your email address in order to invite other AWS accounts to your Organization.
Once verified, click "Add an AWS account" option
Since I have an existing AWS account which I would like to invite in this Organization hence, I will be selecting "Invite an existing AWS account" option. If you do not have an existing account, you can simply create one within the Organization.
Enter the email address or account ID of the AWS account which you want to invite in this Organization
Add message to the owner of the AWS account (optional)
Use tags to associate with the resources
Accept the invitation by clicking on the link sent to the email address OR Login to the member AWS account -> go to AWS Organizations console -> Select Invitations from left pane -> Click Accept Invitation
Now, login back to the Management account and go to AWS Organizations -> AWS Accounts, you should see the member account added to the Organizational Structure Organizational Structure should look like this:



Root
-> management account 
-> member account

Initially, both management and member accounts comes under the Root.

Proceed further to create the groups i.e, OUs within Organization Select the "Root" and go to "Actions" Select "Create New" under Organizational Unit Note: You cannot delete or rename Root. Enter the name for your new OU Mention tags (optional) Click "Create Organizational Unit" at the bottom
Once created, your Organizational Structure should look like this:



Root
-> New OU
-> Management Account
-> Member Account

This shows Root contains one OU named New OU, Management account, and Member account. Currently, New OU is empty.

You can move the member or management accounts to the newly created OU in order to apply SCPs. To move an account from one OU to another, follow the steps below: Select the account which you want to move Go to "Actions" -> "Move" Select the OU under which you want to move that account Click "Move AWS account" at the bottom
I have created the following Organizational Structure:



Root
-> New OU
   -> Member Account
-> Management Account

I have moved member account to new OU and kept Management Account under Root only. Will create SCP to create policies for New OU and member account.

Go to "Policies" in the left pane and Click "Service Control Policies" -> "Enable Service control Policies"
Once enabled, you can create your own SCP policies to apply them on member accounts, or OUs Note: You can apply SCPs to only member accounts in an organization. They have no effect on users or roles in the management account.
To create SCP, follow the below steps:

Go to Policies -> Service Control Policies
Click "Create Policy" on the top right
Fill the details and click "Create Policy" at the bottom
Once created, it should be listed under "Available policies"
Now, select the newly created SCP and attach it to the member accounts or OUs
I have used one of the sample SCPs provided by AWS and attached it to my New OU that contains my member account.

This SCP will deny access to AWS based on the requested AWS Region. Read more details here.

In order to test if the SCP is working or not, I logged into my member account and tried to access the EC2 console. Below is the message displayed on the screen:

It restricted me to launch an EC2 instance and perform any operations which confirms that the SCP is working.

Hope this information helped. Thank you.

Create AWS CloudFormation stack using sample template

Payal Gupta — Sun, 02 Jan 2022 13:31:10 +0000

This post will cover the following content:

What is AWS CloudFormation?
Why should you use it?
How to use it?

Let's get started...

What is AWS CloudFormation?

AWS CloudFormation comes under the AWS M&G category and it helps in provisioning and configuring AWS resources for you.
You just need to create a CloudFormation template to describe what AWS resources you would like to configure along with the properties/settings, and CloudFormation will create them for you as you described.

Why should you use it?

You should use AWS CloudFormation if you are looking to:

Simplify infrastructure management
Instead of creating and managing individual resources in AWS, you can just a CloudFormation stack to easily manage a collection of resources as a single unit in order to save your time and effort.
Quickly replicate your infrastructure
You can reuse your CloudFormation template to provision the same resources in multiple regions without spending time in configuring those resources in each region individually.
Easily control and track changes to your infrastructure
CloudFormation templates are nothing but text files which helps you to track changes to your infrastructure just like version controlling. You can easily roll back to your previous infrastructure using a version control system with templates.

How to use it?

AWS CloudFormation is a regional service and you can use AWS CloudFormation via the browser console, command line tools, or APIs to create stack and resources.

Below are the steps to get started:

Go to the AWS CloudFormation Console
Click on Create stack in the top right corner
Specify the Template.

A template is a JSON or YAML file that contains configuration information about the AWS resources you want to include in the stack.

There are 3 options available for you to choose:

Template is ready - If you have your own template file, then choose this option.
Use a sample template - If you would like to use a sample template file provided by AWS, you should go with this option.
Create template in Designer - If you are new to AWS CloudFormation, you can use this option to create, view, and modify AWS CloudFormation templates using a drag-and-drop interface.

I am selecting a sample template LAMP Stack to begin with CloudFormation.

S3 URL: This is the Amazon S3 bucket URL which is the location of your CloudFormation template file

AWS CloudFormation Sample Template LAMP_Single_Instance: Create a LAMP stack using a single EC2 instance and a local MySQL database for storage. This template demonstrates using the AWS CloudFormation bootstrap scripts to install the packages and files necessary to deploy the Apache web server, PHP and MySQL at instance launch time. WARNING This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.

This is how our template will look like in AWS CF Designer tool

Now, Specify stack details

Stack name: mention a stack name of your choice which can include letters (A-Z and a-z), numbers (0-9), and dashes (-).

Parameters : You can use the given parameters to input customer values while creating or updating the stack.
DBName : Database name for your MySQL DB
DBPassword : Password for your MySQL DB
DBRootPassword : Root password for MySQL DB
DBUser : User name for your MySQL DB access
InstanceType : Specify the EC2 instance type which you would like to use for your WebServer
KeyName : Specify an existing EC2 KeyPair to enable SSH access to the instance.
Make sure you have an existing EC2 keypair created in the specified region. If you do not have one, please create it using the EC2 console and specify the newly created keypair here otherwise stack creation will fail.
SSHLocation : The IP address range that can be used to SSH to the EC2 instances (by default, it is 0.0.0.0/0)

Next, Configure Stack Options

All the given parameters are optional. You can fill them as per your requirements.

Tags : key-value pairs to apply to resources in your stack
Permissions : Specify IAM role name/ARN. By default, CloudFormation uses permissions based on your user credentials.
Stack failure Options : Specify the behavior when stack creation fails
Advanced Options : You can use these options if you would like to define stack policies for designated resources, setup notification, enable stack termination protection, or configure rollback options.

Now comes, Review step. You can review all the details of your CloudFormation stack and if everything looks great, click Create stack at the bottom right corner to initiate stack creation.

PS: At the bottom, there is a Quick-create link option available for you to use. You can create more stacks with the same basic configuration as the one you just created using this URL.

Once the stack creation has completed, you should see the CloudFormation stack status as CREATE_COMPLETE

Resource tab : You should find all the resources created by CF stack under this tab.

Event tab : You can check the CF stack event details under event tab

Outputs : Under this tab, you should see the website URL for your newly created LAMP stack.

Hope you find this information helpful. Thank you.

Connect to the Capella Cluster from Amazon EC2 running Red Hat Enterprise Linux using Couchbase PHP SDK

Payal Gupta — Sat, 01 Jan 2022 19:03:10 +0000

This post will provide you step by step guidelines on how to setup connection with Capella Cluster from Amazon EC2 running Red Hat Enterprise Linux [RHEL 8]

Steps are as follows:

Spin up an EC2 instance running Red Hat Enterprise Linux (ami-0ba62214afa52bec7)

SSH into the EC2 instance. The default user name for a RHEL AMI is ec2-user or root so make sure you are using the correct user name.
$ ssh -i /path/my-key-pair.pem ec2-user@my-instance-public-dns-name

Use below command to install nano editor
$ sudo yum install nano

Install Couchbase C SDK (libcouchbase) which is a pre-requisite to use Couchbase PHP SDK. To do so,
Create a couchbase.repo file in your /etc/yum.repos.d directory.

$ cd /etc/yum.repos.d
$ sudo nano couchbase.repo

paste the below content in couchbase.repo file

[couchbase] enabled = 1 name = libcouchbase package for centos8 x86_64 baseurl = https://packages.couchbase.com/clients/c/repos/rpm/el8/x86_64 gpgcheck = 1 gpgkey = https://packages.couchbase.com/clients/c/repos/rpm/couchbase.key

Now, the repository has been configured, refresh the cache by using below command
$ sudo yum check-update
$ sudo yum search libcouchbase

Install libcouchbase3, and any other packages that you need for development:
sudo yum install libcouchbase3 libcouchbase-devel libcouchbase3-tools

Enable EPEL and Remi Repository

To install EPEL(Extra Packages for Enterprise Linux), run the below command:

$ sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

To install Remi (a third-party repository that provides a wide range of PHP versions for RedHat Enterprise Linux), run below command:

$ sudo dnf install -y https://rpms.remirepo.net/enterprise/remi-release-8.rpm

Install PHP 7.4 on RHEL
First, list the available php module by running this command
$ sudo dnf module list php

You should see remi-7.4 in the list. In order to enable it, run below command:
$ sudo dnf module enable php:remi-7.4 -y

Install the php-devel and php-bear packages
The php-devel package contains the files needed for building PHP extensions and php-pear package contains the basic PEAR components for reusable PHP components.

Run the below commands to install both packages:
$ sudo dnf install php
$ sudo yum install php-devel
$ sudo yum install php-pear

Install the Couchbase PHP SDK using PHP distribution’s pecl command:
$ pecl install couchbase
$ sudo pecl install https://packages.couchbase.com/clients/php/couchbase-3.2.2.tgz

Load the Couchbase SDK as an extension
Use below command to locate the php.ini file
$ php --ini

Insert a line in the php.ini file specifying the extension to be loaded; this should be in the [PHP] section.
extension=couchbase.so

Also, insert the json extension since The Couchbase SDK depends on the JSON module, which must be loaded before the SDK. To do so, you can add the below extension in php.ini file:
extension=json.so

Now, add your ec2 instance public IP address to the allow list of Capella Cluster and run the below sample script to connect with Capella Cluster

<?php
 $connectionString = "couchbases://<connect_string>?ssl=no_verify";
 $options = new \Couchbase\ClusterOptions();
 $options->credentials("<user>", "<password>");
 //$options->timeout(3000 /* milliseconds */);
 $cluster = new \Couchbase\Cluster($connectionString, $options);

 $opts = new \Couchbase\GetOptions();
 $opts->timeout(8000 /* milliseconds */);

 // get a bucket reference
 $bucket = $cluster->bucket("travel-sample");

 // get a default collection reference
 //$collection = $bucket->defaultCollection();

 // or for named collection
 $scope = $bucket->scope("_default");
 $collection = $scope->collection("_default");

 // upsert document
 $upsertResult = $collection->upsert("my-document", ["name" => "mike"]);

 // get document
 $getResult = $collection->get("my-document", $opts);

 echo "Results:\n";
 var_dump($getResult);
?>

where
<connect_string> : Your cluster WAN endpoint
<user>, <password> : your database username/password you created to give access to the cluster buckets
"travel-sample" : It is the sample bucket available in the Capella cluster to use. You could create your own bucket as well. Make sure that your database user has access to that bucket.

Hope the provided steps are helpful. Thank you.

Hello DEV Community!

Payal Gupta — Sat, 01 Jan 2022 12:21:34 +0000

I signed up for a DEV account only to join the exclusive DEV Organisation created for AWS Community Builders to use and publish quality content.

I always wanted to write blogs/articles to share knowledge however, never got enough time to give it a chance and until now, I was only using this platform to check out the great articles written by my co-Community Builders.

After a lot of procrastination, finally, I have decided to start my journey of writing articles/blogs and I would like to give this credit to AWS Community Builder program which pushed and motivate me to give it a shot.

It is an absolute honour to be part of this program where you get the opportunity to learn and grow together by sharing knowledge, networking with great and talented people from all over the world, and helping each other in every possible way.

To give back to the community, I would like to start off by writing a post on the Significance of Migration & Governance on AWS.

PS: I chose this topic because I got selected to be part of this category in AWS Community Builder program so decided to go with it first :)