Forem: Shola Umakhihe

How to build your own LinkedIn Profile Scraper in 2022

Shola Umakhihe — Mon, 08 Aug 2022 13:59:08 +0000

TABLE OF CONTENTS

Introduction
Procedure
- Environment setup
Scraping Samples
The Web Scraping Process
Conclusion

Introduction

LinkedIn is the largest professional network over the internet accessible through mobile or web to look for jobs, internship and enlarge your network. On LinkedIn, you can find people with similar skills, interests, and experience. To access the platform, you need to sign up and create a profile.

On LinkedIn you can search for jobs, internships, and people with similar skills, interests, and experience. What do you say of automating this search process ? This let us to web scraping.

Web scraping is mostly used on sites with big data like Google, Amazon or Twitter. As a whole web scraping is refers to the extraction of data from a website. This information is collected and then exported into a format that is more useful for the user mostly csv file though some other formats are also possible like json.

What follows is tutorial on how to build a scraper in python that can be used to extract data from LinkedIn.

Procedure

Though the scraping is mostly an automation process, it is a broad process that can be broken down into several steps.

Environment setup

Python is the chosen language for this tutorial and as you can guess some precautions should be done to make sure that the environment is setup correctly and the main OS I'll use is Ubuntu(a Linux distro) virtual environment is a tool that helps you to isolate your code from the rest of the system. It is a good idea to create a virtual environment for your project.

Make a new directory on your desktop and cd into it.

# create the directory
mkdir linkedin_scraper

# cd into the directory
cd linkedin_scraper

To create a virtual environment, run the following command:

python -m venv venv
# or
python3 -m venv venv

To activate the virtual environment, run the following command:

# For linux and mac users
source venv/bin/activate

# For windows users
.\venv\Scripts\activate

Install the following packages:

pip install ipython selenium parsel pip-chill

Check the installed packages using the following command:

# To list only the main packages installed in the virtual environment, run the following command:

pip-chill

For this automation process we will use ipython which is a python shell. It is a good idea to use it to run the automation process. On your terminal type the following command:

ipython

Note: Alternatively a jupyter notebook or a python file can be use for this process. ipython was chosen since it is a good shell and it is easy to use with no much prior requirements and interactive results.

Our environment is setup and we're ready to go.

Scraping Samples

To access LinkedIn data we need to login and thus automating this feature too. To automate the login process we will use the selenium package together with the chrome drivers. Follow these commands on your IDE.

ipython
from selenium import webdriver
from selenium.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.linkedin.com")

This will give as result a chrome window having the login page of LinkedIn and most of all it should be having the banner having the message below.

Chrome is being controlled by an automated software.

Next, we need to login to LinkedIn using automation, for this we'll tell our bot it need to provide the login information. For this we'll use the chrome dev tool and get the login fields ids. To access this tool we use the keyboard shortcut Ctrl+Shift+i or we user f12 to open the dev tool.

Now click on the circled Inspect Elements icon, you can hover over any element on the web page the code will appear highlighted as seen above. You want to look for the class and id attributes.

Now click on the circled Inspect Elements icon, you can hover over any element on the web page the code will appear highlighted as seen above. You want to look for the class and id attributes.

From here, you'll be directed to your LinkedIn profile. Guess what, you successfully automated your login process.

Next we want to make a search query on google that will target all the LinkedIn profiles matching the item "Web" AND "JavaScript" on their profile.
Let go to google still using our terminal so that our automated chrome browser will be in use.

# To open the google search page
driver.get('https://www.google.com')

Let's make our query and click on the search button (this is done in the terminal).

search_query = driver.find_element(By.NAME,'q')

search_query.send_keys('site:linkedin.com/in/ AND "Web" AND "Javascript"')

from selenium.webdriver.common.keys import Keys

earch_query.send_keys(Keys.ENTER)

The search can be customised, feel free to modify at your needs.

linkedin_users_urls = driver.find_elements(By.CLASS_NAME, 'iUh30')

Note: The class name is iUh30 and it is the class name of the link that contains the LinkedIn profile URL. Note: The method name we use now is find_elements and it is the method that is used to get all the elements of a certain class. Not the find_element method that get an element.

let's verify that we have some results. We will use the len function to get the length of the list.

len(linkedin_users_urls)

I guess you noticed that the return is not what you wanted. We want to get the URL of the LinkedIn profiles. To get the URL we need to use the get_attribute method and some extra spices. Let's use a new variable to store the URL.

linkedin_users_urls_list = driver.find_elements(By.XPATH, '//div[@class="yuRUbf"]/a[@href]')

# To check the list content we run the following command
[users.text for users in linkedin_users_urls_list]

We got the URL of the linkedin profiles which means we can now start scraping the data(name, title, company, location, the profile URL and more).

The following steps we'll enter a more complex task but we'll use the same methods and variables we used before.

The Web Scraping Process

we need now some files to create our scraper. In your project directory, create the files as follows.

touch variables.py main.py
# Creates two files with the names above

variables.py

my_username = 'your email address'
my_password = 'your passwowrd'

file_name = 'results.csv' # file where the results will be saved

query = 'site:linkedin.com/in/ AND "Web" AND "Javascript"'



> Variables files contains the variables that we'll use to login to LinkedIn together with the query.

We'll use the main.py file to run the main code.

> main.py

 ```python import variables from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.action_chains import ActionChains from selenium.webdriver.common.keys import Keys

driver.get(variables.query) # google search username.send_keys(variables.username) # username field password.send_keys(linkedin_password) # password field linkedin_users_urls_list = driver.find_elements(By.XPATH, '//div[@class="yuRUbf"]/a[@href]') [users.text for users in linkedin_users_urls_list]

This process is fast at times and to slow it down, use the sleep function from the time function and it can used at anytime in the code.

from time import sleep
sleep(2)



The process is summarised as seen below:

1. Login to LinkedIn
2. Making the google search query and submit
3. Getting the different displayed profiles in a list profile_urls
4. Iterate over the list with the new URL as index
5. Get the profile URL and open the profile page
6. Get the name, title, company, location and more from the profile page
7. Save the results to a csv file

We're done.

## Conclusion
As you can see, we've made a scraper that can scrape the data from LinkedIn. We've used the following technologies:
* Selenium
* Python
* Ipython
> Note: From time to time, LinkedIn change their class and attributes, so for future releases, I'll try to update the scraper to work with the new LinkedIn changes. Or you can try to use the LinkedIn API to get the data.

How To Install And Configure The AWS CLI On Linux OS

Shola Umakhihe — Mon, 13 Jun 2022 00:40:24 +0000

The Amazon Web Service Command Line Interface(CLI) is the fastest and easiest way to connect to AWS services and products.

Some instances have the CLI pre-installed such as the Amazon Linux instances while instances like the Red Hat Linux don’t. In this article, I will show you how to install and configure the CLI in an instance.

You’ll need a secure shell connection (SSH), to your AWS instance account and then configure and install.

Pre-requisite: AWS account, SSH Access key, Linux OS installed computer, internet access and a stable mind.

STEP 1: Create an instance in your EC2

- Login to your AWS management console and navigate to EC2
- Create an EC2 with the Red Hat Linux AMI
- Select Network and security in the navigation pane, and enter key pairs
- Select Create Key Pair.
- Then, select the format of the file (. pem)
- Then, download

STEP 2: Connect to your instance using SSH

- Open your terminal with Shift + Ctrl + T
- Navigate to your downloads or anywhere you must have downloaded the .PEM file with “cd downloads”
- Change the permission on it with chmod 400 <filename.pem>
- Go to your AWS management console, go instance.
- Select the the instance type
- By the description tab copy the IPv4 Public IP value.
- Go back to your terminal and run this command ssh -i labsuser.pem ec2-user@<public-ip> (replace <public-ip> with your IPv4 Public IP)
- Type “yes” at the prompt.
- Voila, connected sucessfully

STEP 3: Install the AWS CLI on Red Hat Linux

Now that you have an active SSH connection to the Red Hat Linux instance running on Amazon EC2, next is to install AWS CLI.

- Run sudo yum install -y unzip
- Then, run curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o “awscliv2.zip”
- Then, unzip awscliv2.zip
- Then, sudo ./aws/install
- Verify that AWS CLI is now working by running the following command: aws help
- At the : prompt, type q to exit.
- You are set.

STEP 4: Configure the AWS CLI to connect to your AWS Account

Time to connect your AWS account to your AWS CLI. Go to the active SSH session terminal window and run aws configure. At the prompt enter your AWS Access Key ID and AWS Secret Access Key

We are all set, now you can work on your EC2 instance and every other AWS service from your personal machine using your local terminal.

How to Launch an EC2 Instance on AWS

Shola Umakhihe — Tue, 21 Sep 2021 13:56:07 +0000

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers.

Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change.

Amazon EC2 changes the economics of computing by allowing you to pay only for the capacity that you actually use. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios.

Open Your AWS Management Console
Go to www.console.aws.amazon.com, login to your account, and your management console will open
In the AWS Management Console on the Services menu, click EC2.
Click Launch Instance
Choose an Amazon Machine Image (AMI)

An Amazon Machine Image (AMI) provides the information required to launch an instance, which is a virtual server in the cloud. An AMI includes: A template for the root volume for the instance (for example, an operating system or an application server with applications)
Launch permissions that control which AWS accounts can use the AMI to launch instances
A block device mapping that specifies the volumes to attach to the instance when it is launched

Click Select next to Amazon Linux 2 AMI (at the top of the list).
Choose an Instance Type
Amazon EC2 provides a wide selection of instance types optimized to fit different use cases. Instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources for your applications. Each instance type includes one or more instance sizes, allowing you to scale your resources to the requirements of your target workload.

You will use a t2.micro instance which should be selected by default. This instance type has 1 virtual CPU and 1 GiB of memory. NOTE: You may be restricted from using other instance types in this lab.

Click Next: Configure Instance Details
Choose an Instance Type
Amazon EC2 provides a wide selection of instance types optimized to fit different use cases. Instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources for your applications. Each instance type includes one or more instance sizes, allowing you to scale your resources to the requirements of your target workload.

You will use a t3.micro instance which should be selected by default. This instance type has 1 virtual CPU and 1 GiB of memory. NOTE: You may be restricted from using other instance types in this lab.

Click Next: Configure Instance Details
Configure Instance Details
This page is used to configure the instance to suit your requirements. This includes networking and monitoring settings.

The Network indicates which Virtual Private Cloud (VPC) you wish to launch the instance into. You can have multiple networks, such as different ones for development, testing and production.

For Network, select The Default VPC or Create your own VPC
For Enable termination protection, select Protect against accidental termination.

When an Amazon EC2 instance is no longer required, it can be terminated, which means that the instance is stopped and its resources are released. A terminated instance cannot be started again. If you want to prevent the instance from being accidentally terminated, you can enable termination protection for the instance, which prevents it from being terminated.

Scroll down, then expand Advanced Details.
A field for User data will appear.
When you launch an instance, you can pass user data to the instance that can be used to perform common automated configuration tasks and even run scripts after the instance starts.

Your instance is running Amazon Linux, so you will provide a shell script that will run when the instance starts.
Copy the following commands and paste them into the User data field:
#!/bin/bash
yum -y install httpd
systemctl enable httpd
systemctl start httpd
echo ‘

Hello From Your Web Server!

’ > /var/www/html/index.html

Install an Apache web server (httpd)

Configure the web server to automatically start on boot

Activate the Web server

Create a simple web page

Click Next: Add Storage

Add Storage

Amazon EC2 stores data on a network-attached virtual disk called Elastic Block Store.

You will launch the Amazon EC2 instance using a default 8 GiB disk volume. This will be your root volume (also known as a ‘boot’ volume).

Click Next: Add Tags

Add Tags
Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. This is useful when you have many resources of the same type — you can quickly identify a specific resource based on the tags you have assigned to it. Each tag consists of a Key and a Value, both of which you define.

Click Add Tag then configure

Click Next: Configure Security Group

Configure Security Group
A security group acts as a virtual firewall that controls the traffic for one or more instances. When you launch an instance, you associate one or more security groups with the instance. You add rules to each security group that allow traffic to or from its associated instances. You can modify the rules for a security group at any time; the new rules are automatically applied to all instances that are associated with the security group.

On Step 6: Configure Security Group, configure

Click Review and Launch

Review Instance Launch

The Review page displays the configuration for the instance you are about to launch.

Click Launch

Select an existing key pair or create a new key pair window will appear.

Amazon EC2 uses public–key cryptography to encrypt and decrypt login information. To log in to your instance, you must create a key pair, specify the name of the key pair when you launch the instance, and provide the private key when you connect to the instance.

Click Launch Instances

Your instance will now be launched.

Click View Instances

The instance will appear in a pending state, which means it is being launched. It will then change to running, which indicates that the instance has started booting. There will be a short time before you can access the instance.
The instance receives a public DNS name that you can use to contact the instance from the Internet.

Your Web Server should be selected. The Description tab displays detailed information about your instance.
To view more information in the Description tab, drag the window divider upwards.

Review the information displayed in the Description tab. It includes information about the instance type, security settings and network settings.

Wait for your instance to display the following:

Instance State: running

Status Checks: 2/2 checks passed

Congratulations! You have successfully launched your first Amazon EC2 instance.

OBJECT ORIENTED PROGRAMMING WITH PYTHON

Shola Umakhihe — Wed, 15 Sep 2021 20:04:05 +0000

Object oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviours into individual objects. Conceptually, objects are like the components of a system. Think of a program as a factory assembly line of sorts. At each step of the assembly line a system component processes some material, ultimately transforming raw material into a finished product.

An object contains data, like the raw or preprocessed materials at each step on an assembly line, and behaviour, like the action each assembly line component performs.

FOUR PRINCIPLES OF OOP

The four pillars of object oriented programming are inheritance, encapsulation, abstraction and polymorphism. In inheritance, child classes inherit data and behaviours from parent class. In encapsulation, information are contained in an object thereby exposing only selected information. Abstraction only exposes high level public methods for accessing an object. In polymorphism, many methods can do thesame task.

OOP WITH PYTHON

Object Oriented Programming is a programming paradigm that provides a means of structuring programs so that properties and behaviours are bundled into individual objects. For instance, an object could represent a person with properties like a name, age, and address and behaviours such as walking, talking, breathing, and running or it could represent an email with properties like a recipient list, subject, and body and behaviours like adding attachments and sending.
Put another way, OOP is an approach for modeling concrete, real-world things, like cars, as well as relations between things, like companies and employees, students and teachers and so on. OOP models real world entities as software objects that have some data associated with them and can perform certain functions.

Another common programming paradigm is procedural programming, which structures a program like a recipe in that it provides a set of steps, in the form of functions and code blocks, that flow sequentially in order to complete a task. Objects are at the center of object oriented programming in python.

CLASS

Class is a user defined prototype for an object that defines a set of attributes that characterize any object of the class. The attributes are data members and methods acceseed through dot notation. The class statement creates a new class definition. The name of the class immediately follows the keyword class followed by a colon. Primitive data structures like numbers, strings, and list are designed to represent simple pieces of information, such as the cost of an apple, the name of a poem. Classes are used to create user defined data structures. Classes define functions called methods, which identify the behaviours and actions that an object created from the class can perform with its data.

INSTANCE

An individual object of a certain class is called an instance. An object that belongs to a class is an instance of the class. An instance variable is a variable that os defined inside a method and belongs only to the current instance of a class. Instantiation is the creation of an instance of a class.

OBJECTS

Object is a unique instance of a data structure that’s defined by its class. An object comprises both data members and methods.. To create instances of a classs, you call the class using class name and pass in whatever arguments its init method accepts. You access the object’s attributes using the dot operatot with object. Class variable would be accessed using class name.

Python deletes unneeded objects automatically to free the memory space. The process by which python periodically reclaims blocks of memory that no longer are in use is termed Garbage Collection. Python garbage collector runs during program execution and is triggered when an object’s reference count reaches zero. An object’s reference count changes as the number of aliases that point to it changes. An objects reference count increases whenit is assigned a new name or placed in a container (list, tuple or dictionary). The object’s refernce count decreases when it’s deleted with del, its refernce is reassigned, or its reference goes out of scope. When an object’s reference count reaches zero. Python collects it automatically.

CLASS INHERITANCE

Instead of starting from scratch, you can create a class by deriving it from a preexisting class by listing the parent class in parentheses after the new class name. The child class inherits the attributes of its parent class, and you can use those attributes as if they were defined in the child class. A child class can also override data members and methods from the parent

...TO BE CONTINUED...

Python: The Beginner's Guide

Shola Umakhihe — Wed, 15 Sep 2021 18:10:42 +0000

INTRODUCTION

Python is an open source, high-level programming language developed by Guido Van Rossum in the late 1980s and presently administered by Pythin Software Foundation. It came from the ABC language that he helped create early on in his career. Python is a powerful language that you can use to create games, write GUIs and develop web applications. It is a high level language. Reading and writing codes in python is much like reading and writing regular English statements. Python is an interpreted language which means that every time a program is run, its interpreter runs through the code and translates it into machine-readable byte code. Python is an object oriented language that allows users to manage and control data structures or objects to create and run programs.

Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including structured, procedural, object-oriented and functional programming. Python programs use clear, simple, and concise instructions that are easy to read even by those who have no substantial programming background. Programs written in python are therefore easier to maintain, debug or enhance. Codes used in python are considerably shorter, simpler and less verbose than other high level programming languages . It has a well designed built in features and standard library as well as access to third party modules and source libraries. Python works or runs across all platforms, windows, linux/UNIX, Mac OS, it also works on microcontrollers used in appliances, toys, remote controls, embedded devices, and other similar devices.

GETTING STARTED

Python can be written in any text editor if you have the interpreter also installed. Many developers use special programs that are called Integrated Development Environments (IDEs)/ IDEs help with finding syntax and exception errors. Examples are Microsoft Visual Studio Code, Eclipse, PyCharm, Spyder

To start writing programs in python, you have to first download the installation package compactible with your OS and computer specs from python official website, https://www.python.org/downloads/. After downloading, then install by clicking the .exe file you downlaoded.

Python is a flexible and dynamic language that you can use in different ways. You can use it in script mode when you want to interpret an entire file of statements or application program.

There are several ways to access python’s command line depending on the operating system installed in your computer. If you are using a windows OS, you can start python by clicking on its icon or menu item on the start menu. If you are using GNU/Linux, UNIX, MAC OS systems, you have to run the terminal tool and enter the python command to start your session.

To see how python works, you can use the print command to print the universal program “Yo Good”
Open python’s command line
At the >>>> prompt, type the following: print(“Yo Good”)
Press enter to tell python that you are done with your command. Very quickly, the command line window will display Yo Good on the following line.
To exit from python, you can type any of these commands: quit() or exit() or Ctr+Z+Enter

PYTHON SYNTAX

Python syntax refers to the set of rules that defines how much users and the system should write and interpret a python program. First thing to note are pythin keywords, these are reserved words in python that should not be used as variable, constant, function name, or identifier in your code. Words such as: and, break, continue, global, import, else, except, if, while, from, return, or, etc (check the documentation on www.python.org for more keywords).

PYTHON IDENTIFIER

Next, is identifier; which is a name given to a function, class, variable, module, or other objects that you’ll be using in your python program. Here are python naming conventions you should be aware of:
An identifier can be a combination of uppercase letters, lowercase letters, underscores, and digits (0-9).
Special characters such as %, @ and $ are not allowed within identifiers
An identifier should not begin with a number
Python is a case sensitive language and this behaviour extends to identifiers
Python keywords can’t be used as identifiers
You can use underscores to separate multiple words in your identifier.

PYTHON QUOTATION
Python allows the use of quotation marks to indicate string literals. You can use single, double, or triple quotes but you must start and end the string with thesame type.

PYTHON STATEMENT
Then, we have statements, which are instructions that a python interpreter can execute. When you assign a value to a variable, say pet_var = “lion”, you are making an assignment statement, you just assigned the string “lion” to the variable “pet_var”. In python there are other statements such as the if statements, while statements, etc.

IDENTATION
Python unlike other languages uses indentation to structure its block of codes. Indentation makes python codes more readable and understandable.

COMMENTS
Comments are necessary in every programming language for every programmer as you write your codes. Comments are very handy when you have to review or revisit your program. It will also help others who may come across or look through your code to understand. In python you can use the “#” to start a comment. For multi line comments, you can use the triple quotes.