Forem: Aveek Das

An Overview of AWS IAM Roles and Policies

Aveek Das — Sat, 02 Jul 2022 09:15:27 +0000

In this article, I am going to introduce the concept of AWS IAM, also known as Identity and Access Management in AWS. In any cloud service, controlling who has access to the services and how each of the services accesses the other services is an important task. If we do not control the access or restrict then there might be cases of a security breach within the services and we might not be able to track those as well. So as a best practice to restrict or control access within the AWS, there is a special service called IAM that can be used to manage and control almost everything in AWS. It is the permission control system that controls access to the various AWS resources and services.

In order to follow along with this article, it is best suited that you have an AWS subscription which can be used to verify and understand the topics that we will use in this article.

Understanding AWS IAM components

Identity and Access Management (IAM) is a service offered by Amazon to control the users, create groups and allow or deny access to specific resources within the AWS environment. You can use this service by logging into the AWS Management Console and search for IAM.

The AWS IAM service will be opened and you can see the following. There are four major components of the IAM service.

Users
Groups
Roles
Policies

Let us now understand each of the components in detail.

Users

As per the definition from the official documentation, “An AWS Identity and Access Management (IAM) user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS.”. Basically, you will have to create a user for each and every individual that you would want to access your AWS Account. There is also a possibility to create delegated users for your application, but that is out of scope for this article.

There are two types of users in the AWS Management Console – Root User and IAM User.

The Root user is the first user that is created when you create an AWS Account for the first time. It has access to every possible resource within the AWS Account. As a best practice, it is advised that you should not be using the root user for your daily activities, instead create an IAM user and use it for your development. The root user should only be used to perform specific activities such as billing etc.

The AWS IAM user is one that you create using your root user and can be used to login to the console later. Once the IAM user is created, you should start using it for your daily activities and leave the root user account secured.

Groups

As you might have guessed, in AWS IAM, the groups are a collection of IAM users. Using groups you can get rid of assigning a specific set of permissions to each individual user. Instead, you can assign specific permissions to a group and then add IAM users to that group. All the users in a group will have similar access.

For example, you can consider creating different groups for various teams within your organization like Admins, Developers, Testers, etc. You can then assign specific permissions to these groups. Whenever a new user is onboarded, you can assign the user to a group, and then the user will have all the permissions assigned to that group. You can add the same user to multiple groups. By default, when you create your AWS account, there are no groups created. You need to create groups on your own according to your requirements.

Policies

So far, we have learned about IAM users and groups in AWS, but we did not understand how to allow or deny access to the users or groups. This is specifically done using policies. Policies in IAM are objects that can be assigned to users or groups and control their access to the resources on AWS. Whenever a user requests a specific resource, these policies are evaluated while the request is made. If the policy allows access, then the user or the group will be able to access the specific resource as requested.

There are quite a few types of IAM policies of which the following two are the most important ones.

Identity-based policies – These policies are attached to IAM users or IAM roles (we will see them later in the article). It defines to which resources the IAM user has been granted or rejected access toResource-based policies – These policies are attached to specific resources on AWS to define how they can interact with other resources within the AWS environment. These are used to define delegated permissions for resources like EC2 to have access to an S3 bucket within the same AWS account

A policy in AWS is defined using JSON and looks something like the below.

As you can see in the figure above, the JSON statement defines the administrator policy in the AWS account. There are a few components of the policy JSON statement like version, an array of statements, and each of the statements can have an effect, action, and a resource specified.

Version – It is used to specify the version of the policy language. For most of the cases, you will be using the default and latest version 2012-10-07Statement – Used to define the policy for a specific element. A policy can have one or more statementsEffect – Defines whether you want to allow or deny accessAction – Defines from a list of actions that can be performed on a resourceResource – The IAM ARN of the resource on which the permission is being granted

You can read more about policies in detail from the official documentation. You can also customize your policies to allow more restricted access. For example, you can also define policies that allow access to resources only during specific hours of the day or only while using a Multi-Factor Authentication. These are complex topics and beyond the scope of this article.

Roles

IAM Roles are an important concept in AWS. You can think of roles as permissions, not for a user, instead of for an AWS service itself. In the previous sections of the article, we have seen what IAM users are and how to allow access to these users to specific resources in AWS. However, when an AWS resource needs to access contents within the AWS, then we cannot create a user for the user. In such cases, we create roles with policies and then allow the AWS resource to assume the role while making the requests.

You can consider a scenario where an EC2 instance might want to read data from an S3 bucket. Here, we can create a role that has FullS3Access and then allow the EC2 instance to assume the role.

Conclusion

In this article, we have understood the concept of AWS IAM. It is one of the most important services that need to be correctly set up for any organization before the development and other activities are initiated. There are also a few best practices suggested by Amazon that should be followed while designing the IAM structures. As already mentioned in the article, there are two ways of accessing resources on AWS, either by a username and password or by using secret access keys. You should also keep an active eye on and monitor the activities within your AWS Account. You can read more about monitoring activities from the official documentation.

Use Cloud Formation Templates to spin up MySQL instances on RDS

Aveek Das — Tue, 22 Mar 2022 08:27:05 +0000

In this article, I am going to discuss how to set up a MySQL instance on AWS RDS using Cloud Formation templates. In my previous article, How to configure an Amazon RDS environment for MySQL, I have provided a detailed walkthrough of how to set up a MySQL instance on Amazon. You can use the AWS console to provide all the information required for setting up the instance and then use it. However, in this article, we will discuss an automated way of achieving the same functionalities using Cloud Formation templates.

What is a CloudFormation template?

In order to start writing the templates, we first need to understand what cloud formation is all about. AWS offers Cloud Formation as a cloud service that enables AWS customers to write the desired state of any infrastructure as a code and then use it to deploy resources to AWS. It allows us to create a resource or a group of resources by following simple configuration templates. For example, if you want to deploy a web application on an EC2 instance that will also use an RDS database, you can simply combine both the resources into one stack and write the code for setting up the instances. Once you run, it will create both the EC2 and the RDS instance and also deploy your web application.

Figure 1 – CloudFormation Template on AWS Console

The components of a cloud formation template are as follows:

A JSON or YAML file in which the resources will be defined as a template
A stack can be a combination of multiple resources that needed to be set up as a part of the application. For e.g., EC2 and RDS
A changeset can be used to view a list of operations to be performed by the stack
A stack set also can be considered as a managed group of stacks that be reproduced or replicated

Out of the above, we will mainly focus on the JSON/YAML template and the stack in this article. The templates support a wide range of resources on AWS. To get an exhaustive list please follow this link.

Writing the JSON or YAML template file

One of the main challenges that we face while writing a template for the first time is what contents should be included in it. To ease that AWS provides a lot of template samples that can be easily found on the official documentation. Once you find the template required to create a necessary resource, you can use it to create your own template. The template can either be created using JSON or YAML and converted from one to another as well. On a personal note, I prefer writing the templates using YAML as it helps me keep away from the curly braces and commas and keeps the document cleaner. However, you are free to choose any option between JSON and YAML.

For this exercise, let us try to spin up a MySQL instance on RDS and we will write the code in YAML. In order to set up a MySQL database using cloud formation, we would need the following items.

Resource Name – The name of the resource you are going to set up.
Resource Type – The type of the resource.
Resource Properties – The database name, username, and password to connect to.
The database Engine – MySQL, SQL Server, PostgreSQL, etc.
Storage Type – The storage type to be used.
Public Accessibility – Weather the database should have public access or not.
Allocated Memory – The memory that should be allocated to the database instance.
AWS Region – The region in which the instance is to be created.

Now that we have some points with us, let us go ahead and start by writing the template file.

https://gist.github.com/aveek22/a188b02abde65479bbba2deba5ec90d6

Figure 2 – The template file for creating MySQL instance on RDS

As you can see in the figure above, the structure of the file is divided into two parts. In the first part, we have defined a few parameters that will be used by the resource definitions in the second part while creating the resources on AWS. You can use the same script on your machine to create an instance on your AWS account.

Setting up the stack on the console

Now that we have created our template for MySQL, it’s time we set it up to see if everything runs well. Head over to the AWS console and search for Cloud Formation on the search bar. Click on Create Stack and upload the template file from your local. Click Next once done.

Figure 3 – Creating the stack from the template

On the next page, you will be asked to provide the parameters that were defined in the cloud formation template. Go ahead and provide the necessary details and click Next.

Figure 4 – Stack Parameters provided

Review the stack changes and parameters and click on Create Stack on the last page. The stack creation will start. It might take some time to create all the resources before they are available to use. In the meantime, you can click on the Reload button to check if all the events have been created successfully.

Figure 5 – Stack Creation in progress

As you can see on the timestamps, it took me around 6 minutes after which the resources were available and were also visible on the console. Navigate to the Resources tab and click on the Physical ID of the resource. This will take you to the RDS console.

Figure 6 – Stack Created Successfully

Connecting the MySQL database

Once you are in the RDS console, you can get the hostname for the database instance and use it to connect to the database. Grab the endpoint from the console and use it to connect.

Figure 7 – MySQL instance details on RDS

Let us now head over to MySQL Workbench and try connecting to the hostname with the credentials that we have provided earlier. Use the hostname, username, and password to connect to the instance.

Figure 8 – Connecting to the MySQL instance using MySQL Workbench

As you can see in the figure above, we have successfully connected to the database instance on RDS. Let us now run some queries and see if it works.

Figure 9 – Executing scripts in the MySQL Workbench

As you can see in the figure above, we have been able to connect to the database instance and execute queries. With this, you are able to write your own cloud formation template and spin up any other resources like the SQL Server or PostgreSQL database.

Removing the resources

If you are performing this as an exercise, I would recommend removing the resources that we have created as it will incur some charges. You can navigate to the RDS console and then select the instance and click on Delete to remove the resources permanently.

Figure 10 – Deleting the resources created

Once the resources have been deleted successfully, you can check in the console once again if there are any instances running just to avoid adding unnecessary billing costs.

Conclusion

In this article, we have discussed in detail how to set up a MySQL database instance in an AWS RDS environment and automate it using the Cloud Formation template. This way if setting up resources on the cloud is also known as infrastructure-as-code service where you can set up the entire infrastructure as code and then use it in various environments. These templates can be used not only to deploy database instances but also other infrastructures within the AWS such as EC2, Lambda, RedShift, etc. To learn more about the cloud formation templates, you can view the sample templates available on the official website.

An introduction to sharing content on LinkedIn using REST APIs and Python

Aveek Das — Mon, 14 Mar 2022 11:02:34 +0000

Introduction

LinkedIn is a popular social media platform that allows users to connect to each other and maintain professional network. You can manage your profile and also create a company profile for your organization. An interesting feature of LinkedIn is the ability to share posts on your feed or on your organization’s feed. You can embed URLs, provide hashtags and emojis and also share images and videos in your post. Although the primary way of sharing content on LinkedIn is by using the web application, you can also do it programmatically using the REST APIs provided by LinkedIn and a programming language of your choice.

In this article, we are going to learn how to use the LinkedIn REST APIs and share content from your personal profile as well as from an organization’s profile that you manage. Don’t worry if you do not have an organization as I will guide you from the scratch.

Pre-requisites

In order to proceed further, we need to have the following pre-requisites.

LinkedIn Account – You need to have a LinkedIn account, if you do not have one, please head over to https://www.linkedin.com/ and create one.
REST APIs – Some basic knowledge of using REST APIs.
Python – General programming knowledge of using python and using REST APIs in python.

Apart from the above-mentioned, I am going to use Jupyter notebooks to create the codebase and make it available using GitHub. At any point in time, you can choose to refer to the official documentation of LinkedIn from Microsoft, however, I will try to explain the process in simple terms.

Creating the LinkedIn Company Page and Application

Assuming that you already have your LinkedIn account created, now we will create the following two assets.

LinkedIn Company Page – Also, can be referred to as the Company page, we need to create it using your personal account. You will be an admin for the Company page that you create.
An Application on the Developers portal – In order to get access to the LinkedIn REST APIs, we need to create an application, that will assume the permissions on our behalf and share content on your profile or from the organisation’s profile.

Create the LinkedIn Company Page.

Let’s create a company page from your LinkedIn account. You can think of any dummy name for your company to proceed forward. I am going to use my company as “DataScholar”.

Step 1: Open LinkedIn account. Click on Work and select Create a Company Page. Click on Company and enter the details and click on Create Page.

Figure 1 – Create LinkedIn company page

Step 2: You need to provide the following details for your company page to be created.

Name – The name of your company. I am using DataScholar.
Company URL – The LinkedIn username of your company.
Website – The website your company is hosted on. I am going to use “https://thedatascholar.com”.
Industry – Select as Software.
Company Size – I have selected between 0 and 1 since I am the sole user of the company.
Company Type – Non-profit.
Logo – Any custom image for your company logo.
Tagline – A short description of your company.

Once you have provided all these details, your LinkedIn company page has been created now.

Figure 2 – LinkedIn company page

Create the LinkedIn app in the Developer Portal.

In this section, we will dig a bit deeper and create an application on LinkedIn Developer’s platform that will allow us to interact with the LinkedIn assets programmatically.

Step 1: Navigate to https://developer.linkedin.com/. Click on My Apps and then select Create App. Enter the details for your application.

App Name – The name for your application. You can use the same name as your company.
LinkedIn Page – The company page that the app will be associated with.
Privacy Policy URL – Optional field.
App Logo – Upload a logo for your app.

Agree to the terms and then click on Create App.

Figure 3 – Creating LinkedIn App

Step 2: Click Verify to verify your application.

Figure 4 – Verify LinkedIn application
Step 3: Click Generate URL and copy the URL and click I’m Done.

Figure 5 – Generate verification URL for the LinkedIn app
Step 4: Copy and hit the URL on the browser. It will ask you to verify the app for the page.

Figure 6 – Verify LinkedIn app and Company association
Step 5: Click on Verify.

Figure 7- LinkedIn app and Company association verified
Great! You have now successfully created a LinkedIn company page and also created an app to interact with LinkedIn using the REST APIs.

Configure your LinkedIn App

Let’s configure the LinkedIn app that we have created in the previous section. We will add the products required to access the LinkedIn REST APIs and store the app credentials.

Add the products for your LinkedIn App.

Once you have created the LinkedIn App, you need to define what are the products that you are going to use with your app. Based on this, your app will be provided with the necessary permissions that you need. There are three products available to choose from.

Share on LinkedIn – This allows you to post content from your profile.
Sign In with LinkedIn – This allows you to use LinkedIn social sign on your webpage.
Marketing Developer Platform – Allows you to post content from your page.

As a part of this article, I am going to add all three products to my app.

Figure 8 – Add products to LinkedIn app
The Marketing Developer Platform requires additional access. You will receive a form for this. Fill up the form and wait for approval (2 business days). Click on the View Access Form and fill it up with relevant details.

Figure 9 – Fill up review form for Marketing Developer Platform
Once you submit the form, it might take up to 2 business days to get an answer by email. You will then notice the app has been added to your product list.

💡 The Marketing Developer Platform allows you to post content from your company page. Without adding it, you can still post content from your personal LinkedIn profile.

Fetch and store the app credentials.

Step 1: On your LinkedIn app, click on the Auth tab. You can find the Client ID and Client Secret that you need later.

Figure 10 – Store LinkedIn app credentials
Step 2: Add the redirect URL for your App. This can simply be your webpage URL appended with “/auth/linkedin/callback”.

Figure 11 – Setup app Redirect URI
Step 3: Get the Organization ID from the company URL.

Figure 12 – Get Organization ID

Start assembling your Python code

Now let’s start building the python program that would allow us to interact with the LinkedIn REST APIs.

Setup static variables and libraries

In order to work with REST APIs in python, we will leverage the requests library. You can install it using the python pip manager. In addition to that, we will also create variables to store our client_id, client_secret, and organization_id that we have obtained in the previous section.

In the next steps, we need to get an Access Token that can be used to share content from the LinkedIn profile. The Access Token can be obtained in two steps as follows.

A GET request that will ask the user to allow your app to obtain the required permissions and redirect you to a new URL. From this URL, you’ll need to extract the value in the parameter code.
A POST request that will return your brand new Access Token.

Create the GET request to obtain verification code

Setup the following details to build the URL.

base_url: The LinkedIn endpoint to obtain authorization code.
redirect_uri: The redirect URL you have set while creating your app.
scope: The scope required to publish content from your personal profile.
state: A random text that will be returned to you after authorization. This allows you to verify the response from CSRF attacks. For simplicity, let’s not bother much about it now.

.gist table { margin-bottom: 0; }

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

Show hidden characters


	base_url = "https://www.linkedin.com/oauth/v2/authorization"
	redirect_uri = "https://thedatascholar.com/auth/linkedin/callback"
	scope = "w_member_social,r_liteprofile"

	url = f"{base_url}?response_type=code&client_id={client_id}&state=random&redirect_uri={redirect_uri}&scope={scope}"
	print(url)

view raw
social_linkedin_auth.py
hosted with ❤ by GitHub

When you run the above code, it will print the URL on your console. Copy the URL and hit it in your browser. You will be asked to log in with your LinkedIn credentials. Once your login is verified, it will redirect you to the address mentioned in the redirect_uri. You will also receive two parameters, code, and state in this URL. Copy the value of the code and store it for processing the next request.

💡 To publish content from your company page, you need more permissions in your scope. This can be obtained once you have received verification for the Marketing Developer Platform from the LinkedIn team.

Create the POST request to obtain the Access Token

Now that the verification code has been obtained, we can get the access token by sending a POST request to LinkedIn. You need to provide the following as a payload in the POST request.

grant_type: The grant type should be ‘authorization_code’.
code: The authorization code obtained in the previous step.
redirect_uri: The redirect URI of your app.
client_id: The client ID that we stored earlier.
client_secret: The client secret that we stored earlier.

.gist table { margin-bottom: 0; }

Show hidden characters


	url_access_token = "https://www.linkedin.com/oauth/v2/accessToken"
	auth_code = "YOUR_AUTH_CODE_FROM_PREVIOUS_VERIFICATION"

	payload = {
	'grant_type' : 'authorization_code',
	'code' : auth_code,
	'redirect_uri' : redirect_uri,
	'client_id' : client_id,
	'client_secret' : client_secret
	}

	response = requests.post(url=url_access_token, params=payload)
	response_json = response.json()

	# Extract the access_token from the response_json
	access_token = response_json['access_token']

view raw
social_linkedin_access_token.py
hosted with ❤ by GitHub

If the POST request is successful, you will receive a JSON response that will contain the access_token. I have extracted and stored it in a separate variable.

Get your LinkedIn profile ID

In order to post content from your LinkedIn profile, you need to obtain your profile ID. This can be done by querying the LinkedIn REST API directly.

.gist table { margin-bottom: 0; }

Show hidden characters


	# Get LinkedIn user ID
	url = "https://api.linkedin.com/v2/me"

	header = {
	'Authorization' : f'Bearer {access_token}'
	}

	response = requests.get(url=url, headers=header)
	response_json_li_person = response.json()

	person_id = response_json_li_person['id']

view raw
social_linkedin_profile.py
hosted with ❤ by GitHub

Notice that I have passed the access_token as a bearer token in the authorization header. Also, I have stored the LinkedIn profile ID in a variable called person_id.

Share content from LinkedIn Personal Profile

Let’s prepare the content that we want to share from our LinkedIn personal profile. I have selected a random article from Redhat that I would like to share. Also, I would like to add a thumbnail image to the content that is shared. For that, you can choose any image that is publicly available. Notice that the payload contains the key owner and the value is the profile ID that we obtained in the earlier step. This value determines if the post is to be shared from the personal profile or from the company page.

.gist table { margin-bottom: 0; }

Show hidden characters


	url = "https://api.linkedin.com/v2/shares"

	headers = {
	'Authorization' : f'Bearer {access_token}',
	'Content-Type' : 'application/json'
	}

	payload = {
	"content": {
	"contentEntities": [
	{
	"entityLocation": "https://www.redhat.com/en/topics/api/what-is-a-rest-api",
	"thumbnails": [
	{
	"resolvedUrl": "https://images.pexels.com/photos/2115217/pexels-photo-2115217.jpeg"
	}
	]
	}
	],
	"title": "What is a REST API?"
	},
	'distribution': {
	'linkedInDistributionTarget': {}
	},
	'owner': f'urn:li:person:{person_id}',
	'text': {
	'text': f'Learn more about REST APIs in details. \n#restapi #api'
	}
	}

	response = requests.post(url=url, headers=headers, json = payload)

	print(response.json())

view raw
social_linkedin_share_post_personal.py
hosted with ❤ by GitHub

If you want to understand the payload in more depth, please refer to the official documentation. There is a clear explanation of what each of the fields in the payload is about. Once the request is successful, the content will be shared from your personal profile and it will appear in the feed as follows.

Figure 13 – Post shared from the personal profile on LinkedIn

Share content from LinkedIn Company Page

Sharing content from an organization’s page is almost similar to that of sharing from a personal profile. The only key that determines whether the post content is to be shared from a personal or organization’s profile is the owner. The following values are accepted for the owner key.

Personal Profile: urn:li:person:{person_id}
Organization: urn:li:organization:{organization_id}

You can fetch the organization_id from the URL of your company page. With that, now the payload can be modified and content shared from the organization’s profile.

.gist table { margin-bottom: 0; }

Show hidden characters


	url = "https://api.linkedin.com/v2/shares"

	headers = {
	'Authorization' : f'Bearer {access_token}',
	'Content-Type' : 'application/json'
	}

	payload = {
	"content": {
	"contentEntities": [
	{
	"entityLocation": "https://www.redhat.com/en/topics/api/what-is-a-rest-api",
	"thumbnails": [
	{
	"resolvedUrl": "https://images.pexels.com/photos/2115217/pexels-photo-2115217.jpeg"
	}
	]
	}
	],
	"title": "What is a REST API?"
	},
	'distribution': {
	'linkedInDistributionTarget': {}
	},
	'owner': f'urn:li:organization:{organization_id}',
	'text': {
	'text': f'Learn more about REST APIs in details. \n#restapi #api'
	}
	}

	response = requests.post(url=url, headers=headers, json = payload)

	print(response.json())

view raw
social_linkedin_share_post_organization.py
hosted with ❤ by GitHub

This will post the content from your organization’s profile and you will have it on your feed.

Figure 14 – Post shared from organization’s profile on LinkedIn
💡 To post content from the organization’s profile, you need to wait until you have received the email from LinkedIn which might take 2 business days.

Conclusion

In this article, we have focused on how to use the LinkedIn REST APIs and share content from a personal profile as well as from an organization’s profile. This is just a beginner level introduction to using the LinkedIn REST APIs. Apart from sharing content, you can also use other products to interact with LinkedIn Ads, and Campaign Management Tools.

Resources

If you want to learn more about LinkedIn REST APIs, I would recommend reading through the following.