Forem: AC Nice

How to create LinkedIn Job API using Python

AC Nice — Tue, 02 Aug 2022 23:30:19 +0000

Content

Intro
Requirements
What you can use LinkedIn Jobs API for
Getting Started
Install the Python libraries
Quick Usage From Python Interpreter
Job Search API
Jobs API

Intro

Creating your own LinkedIn Job RESTful API can be an amazing way to build customized job openings applications, or even build a service around the data you’ve collected, or it can fairly be a fun personal project that allows you to learn a new skill.

An API is an acronym for Application Programming Interface.

It is a set of functions or procedures that serves as an interface for connecting an application's services, data and functionalities with other independent service providers, operating systems etc.

While this may seem like an intimidating task at catch of a glimpse, it’s actually in every aspect easy.

Requirements

To gain access to this API, we will need:

A LinkedIn Account and Working page
Python 3 programming language
A terminal or Code editor

The LinkedIn API provides a service that lets people take their LinkedIn profiles and networks with them to other sites or applications via an OAuth based interface.

What you can use LinkedIn Jobs API for

The LinkedIn Jobs API can be applied in the following cases and more:

Job postings crawling
Create and manage jobs
Promote jobs
Manage jobs lifecycle
Job openings and career analysis and many more.

By the end of this tutorial you will be able to do and understand the followings:

How to get access to LinkedIn Jobs API
Install the python3-LinkedIn library for working with LinkedIn APIs
Perform LinkedIn Developer Authentication
Fetching a list of jobs.
Getting job details
Discover the limitations of LinkedIn Jobs API and more.

Getting Started

To get started, we first need to acquire an API secret key from LinkedIn. This is very easy to do! 😎. By following the steps below, you should be able to get your LinkedIn API credentials.

Visit the LinkedIn Developer web page

Click on the create app

It’s Important you have a LinkedIn page associated with the app. In case you don’t have any page on your LinkedIn, don’t sweat it, you can create one by clicking on the Create a new LinkedIn page button below.

Click on Auth and under the Application credentials card you will find your Authentication keys.

Easy isn’t it? 😀

Now, you have been able to acquire your API secret keys, next is to install necessary python libraries that will aid in this our operation.

Install the Python libraries

Run the code below on your terminal or command prompt to install the python3-linkedin library.

pip install python3-linkedin

The library can as well be downloaded and compiled using the following commands:

## First clone the repository
git clone https://github.com/DEKHTIARJonathan/python3-linkedin.git

## Then install the library
python setup.py install

This library provides an interface to the LinkedIn Profile, Group, Company, Jobs, Search, Share, Network and Invitation REST APIs.

The LinkedIn REST API supports the OAuth 2.0 protocol for authentication.

Quick Usage From Python Interpreter

To quickly test the library using an python interpreter, you can do that comfortably from the test server.

from linkedin import server
application = server.quick_api(KEY, SECRET)

This will print the authorization url to the screen. Go into that URL using a browser to grant access to the application. After you do so, the method will return with an API object you can now use.

Job Search API

The Job Search API enables search across LinkedIn’s job postings. You can get more information from here.

application.search_job(selectors=[{'jobs': ['id', 'customer-job-code', 'posting-date']}], params={'title': 'python', 'count': 2})
{u'jobs': {u'_count': 2,
  u'_start': 0,
  u'_total': 206747,
  u'values': [{u'customerJobCode': u'0006YT23WQ',
    u'id': 5174636,
    u'postingDate': {u'day': 21, u'month': 3, u'year': 2013}},
   {u'customerJobCode': u'00023CCVC2',
    u'id': 5174634,
    u'postingDate': {u'day': 21, u'month': 3, u'year': 2013}}]}}

Jobs API

The Jobs APIs provide access to view jobs and job data.

application.get_job(job_id=5174636)
{u'active': True,
 u'company': {u'id': 2329, u'name': u'Schneider Electric'},
 u'descriptionSnippet': u"The Industrial Accounts Sales Manager is a quota carrying senior sales position principally responsible for generating new sales and growing company's share of wallet within the industrial business, contracting business and consulting engineering business. The primary objective is to build and establish strong and lasting relationships with technical teams and at executive level within specific in",
 u'id': 5174636,
 u'position': {u'title': u'Industrial Accounts Sales Manager'},
 u'postingTimestamp': 1363860033000}

You can also fetch you job bookmarks.

application.get_job_bookmarks()
{u'_total': 0}

Sales Trend Analysis With Pandas

AC Nice — Wed, 02 Feb 2022 18:28:18 +0000

Assessment

You have just been employed as Data analyst in one of the fast growing product manufacturing and distribution companies and your first welcoming task by MD is to create a report for an upcoming board meeting. You are to go through and analyze the sales data from 2015-2017 in order to generate the requested report.

The report should capture the following;

Revenue by region
Revenue by sales Rep
Revenue by products
Sales trend
Yearly changes in revenue Highlight the following on the report:
Top 3 products
The most productive sales Rep in the respective years. From your analysis, give 3 recommendations you think would help the company increase revenue in the following year.

Data Source:

https://docs.google.com/spreadsheets/d/1SWCOO70Yv7PPvmlLyLQj1gCilg1XsSl0/edit?usp=sharing&ouid=102731908679789782364&rtpof=true&sd=true

Github code: https://github.com/anochima/Sales-Data-Analysis



import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns



# Setup our visuals to use seaborn by default
plt.style.use('seaborn')
plt.rcParams["figure.figsize"] = (11, 4)



# Import our data using pandas
df = pd.read_excel('MODULE 2-Assessment working data.xlsx')

# Convert the sales excel file into .csv (Comma Separated Value)
# This is only done because because I feel more comfortable working with Csv files to xlsx files
df = df.to_csv('sales-data.csv', index=False)

# Read the sales data
df = pd.read_csv('sales-data.csv')



df.head()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Date	SalesRep	Region	Product	Color	Units	Revenue
0	2015-11-06	Julie	East	Sunshine	Blue	4	78.8
1	2015-11-07	Adam	West	Bellen	Clear	4	123.0
2	2015-11-07	Julie	East	Aspen	Clear	1	26.0
3	2015-11-07	Nabil	South	Quad	Clear	2	69.0
4	2015-11-07	Julie	South	Aspen	Blue	2	51.0



df.describe(include=['object','float','int'])

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Date	SalesRep	Region	Product	Color	Units	Revenue
count	9971	9971	9971	9971	9971	9971.000000	9971.000000
unique	717	6	3	7	5	NaN	NaN
top	2016-09-11	Julie	West	Bellen	Red	NaN	NaN
freq	77	2233	4417	1948	3020	NaN	NaN
mean	NaN	NaN	NaN	NaN	NaN	3.388828	91.181513
std	NaN	NaN	NaN	NaN	NaN	4.320759	120.894473
min	NaN	NaN	NaN	NaN	NaN	1.000000	21.000000
25%	NaN	NaN	NaN	NaN	NaN	2.000000	42.900000
50%	NaN	NaN	NaN	NaN	NaN	2.000000	60.000000
75%	NaN	NaN	NaN	NaN	NaN	3.000000	76.500000
max	NaN	NaN	NaN	NaN	NaN	25.000000	1901.750000

Some useful Insights

There was a total of 9,971 sales entries between 2015-2017 out of which the following descriptions were made:

Units:

The minimum number of units sold between 2015-2017 was 1
The maximum number of units sold between 2015-2017 was 25
The average number of units sold between 2015-2017 was aproximately 3

Revenue

The least revenue generated between 2015-2017 was 21
The most revenue between 2015-2017 was approximately 1902

---Others---

We had 6 unique Sales Representatives between 2015-2017
We covered 3 Regions between 2015-2017
5 unique colors



# At what amount did we sell most?

df['Revenue'].value_counts().hist(bins=50);

Here most items were sold between 21 - 70 respectively



df['Units'].hist(bins=50);



# What's the total revenue generated between 2015-2017?

round(df['Revenue'].sum())



df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 9971 entries, 0 to 9970
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Date      9971 non-null   object 
 1   SalesRep  9971 non-null   object 
 2   Region    9971 non-null   object 
 3   Product   9971 non-null   object 
 4   Color     9971 non-null   object 
 5   Units     9971 non-null   int64  
 6   Revenue   9971 non-null   float64
dtypes: float64(1), int64(1), object(5)
memory usage: 545.4+ KB



# Check if we have any missing entry
df.isna().sum()

Date        0
SalesRep    0
Region      0
Product     0
Color       0
Units       0
Revenue     0
dtype: int64

No entry is missing

Revenue by region



region_revenue = pd.DataFrame(df.groupby(by=['Region'])['Revenue'].sum())
region_revenue.sort_values(ascending=False, by='Revenue')

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Revenue
Region
West	408037.58
South	263256.50
East	237876.79

Visualize region revenue impact



region_revenue.plot(kind='bar', ylabel='Revenue', title='Region revenue impact');

It's very clear that the West Region generated the most revenue

Revenue by sales Rep



sales_rep_revenue = df.groupby(by=['SalesRep'])['Revenue'].sum()
sales_rep_revenue = pd.DataFrame(sales_rep_revenue).sort_values(ascending=True, by='Revenue')
sales_rep_revenue

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Revenue
SalesRep
Nicole	92026.68
Adam	102715.60
Jessica	145496.28
Nabil	158904.48
Julie	204450.05
Mike	205577.78

Visualize salesRep revenue impact



sales_rep_revenue.plot(kind='bar', ylabel='Revenue', title='SalesRep revenue impact');



print('Mike Slightly beat Julie in revenue generation by ' + str(round(((sales_rep_revenue.loc['Mike'] - sales_rep_revenue.loc['Julie']) / sales_rep_revenue.loc['Julie']) * 100, 2))+'%')

Mike Slightly beat Julie in revenue generation by Revenue    0.55
dtype: float64%

Revenue by products



product_revenue = df[['Units', 'Revenue','Product']].groupby('Product').sum().sort_values(ascending=False,by='Units')
product_revenue

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Units	Revenue
Product
Bellen	6579	168175.05
Quad	6223	194032.15
Sunbell	4500	114283.09
Carlota	4371	101272.05
Aspen	4242	96382.80
Sunshine	4229	85983.80
Doublers	3646	149041.93

Visualize of Products revenue impact



product_revenue.groupby(by=['Product'])['Revenue'].sum().sort_values(ascending=True).plot(
                                                                                          kind='bar',ylabel='Revenue',title='Product Revenue');

Sales trend



# Convert the date column to a datetime object
df['Date'] = pd.to_datetime(df['Date'])
df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['Day']  = df['Date'].dt.day
# df = df.drop('Date',axis=1)

Plot Yearly Sales Chart/Trend



years = [unique for unique in df.Year.unique()]
years

[2015, 2016, 2017]



def plot_trend(years:list, df):
    for year in years:
        new_df = df[df['Year'] == year]
        new_df.groupby('Date')['Revenue'].sum().plot(linewidth=1.2, 
                                             ylabel='Revenue', 
                                             xlabel='Date', 
                                             title='Sales Trend');



import matplotlib.patches as patches

year1 = patches.Patch(color='blue', label='2015')
year2 = patches.Patch(color='green', label='2016')
year3 = patches.Patch(color='red', label='2017')
plot_trend(years, df)
plt.legend(handles=[year1,year2,year3], loc=2);

The trend plot looks symmetrical for the months of October in 2017 and 2018 respectively.
This shows that most sales are made within the month of October. what could have influenced this?



ax = df[['Month', 'Units', 'Revenue']].groupby('Month').sum().plot(
                                                                 title='Monthly Sales Trend', 
                                                                 ylabel='Revenue',
                                                                 );
ax.vlines(10,1,300000, linestyles='dashed')
ax.annotate('Oct',(10,0));

How many times was entry made in each month?



df['Month'].value_counts().sort_values().plot(kind='bar', xlabel='Month', ylabel='Number of Entries', title='Monthly Entries');



products = pd.DataFrame(df[['Units','Revenue','Product','Month', 'Region']].groupby('Month')['Product'].value_counts())
products

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

		Product
Month	Product
1	Bellen	52
	Quad	46
	Sunbell	34
	Aspen	33
	Sunshine	33
...	...	...
12	Sunbell	43
	Aspen	41
	Sunshine	36
	Carlota	35
	Doublers	32

84 rows × 1 columns



products['No_of_products'] = products['Product']
products.drop('Product', inplace=True, axis=1)
products = products.reset_index()



products

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Month	Product	No_of_products
0	1	Bellen	52
1	1	Quad	46
2	1	Sunbell	34
3	1	Aspen	33
4	1	Sunshine	33
...	...	...	...
79	12	Sunbell	43
80	12	Aspen	41
81	12	Sunshine	36
82	12	Carlota	35
83	12	Doublers	32

84 rows × 3 columns



products = products.pivot_table(values=['No_of_products'], index=['Month'], columns=['Product'], aggfunc= np.sum)
products

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead tr th {
    text-align: left;
}

.dataframe thead tr:last-of-type th {
    text-align: right;
}

	No_of_products
Product	Aspen	Bellen	Carlota	Doublers	Quad	Sunbell	Sunshine
Month
1	33	52	30	29	46	34	33
2	26	45	29	25	45	35	26
3	49	55	46	34	77	35	39
4	31	37	35	27	50	43	25
5	33	52	36	30	51	37	31
6	37	53	23	21	52	30	36
7	38	60	37	25	55	46	42
8	34	54	26	35	51	35	35
9	380	596	399	311	560	397	404
10	476	735	439	333	697	491	460
11	119	154	118	77	142	108	103
12	41	55	35	32	64	43	36



products.plot(ylabel='No of Products sold', title='Monthly product sales');

It's clear now that we sell more Bellen during october and that and HEAVY sales kick off between September and late November

Region Monthly Revenue



region_sales = pd.DataFrame(df[['Units','Revenue','Product','Month', 'Region']]).groupby(['Month','Region'])['Revenue'].sum()
region_sales = pd.DataFrame(region_sales)
region_sales

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

		Revenue
Month	Region
1	East	5012.34
	South	7551.55
	West	8550.33
2	East	6428.75
	South	5540.10
	West	10864.87
3	East	6082.75
	South	8863.80
	West	14087.99
4	East	6420.63
	South	7647.28
	West	8865.57
5	East	8782.68
	South	5651.30
	West	10962.00
6	East	6442.85
	South	3954.90
	West	9020.65
7	East	7180.45
	South	10155.59
	West	10150.25
8	East	6031.55
	South	7767.60
	West	11567.37
9	East	70532.44
	South	83228.39
	West	127160.06
10	East	87858.60
	South	92034.70
	West	151780.43
11	East	19478.10
	South	24048.59
	West	33196.52
12	East	7625.65
	South	6812.70
	West	11831.54



region_sales = region_sales.reset_index()
region_sales = region_sales.pivot_table(values=['Revenue'], index=['Month'], columns=['Region'], aggfunc= np.sum)
region_sales

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead tr th {
    text-align: left;
}

.dataframe thead tr:last-of-type th {
    text-align: right;
}

	Revenue
Region	East	South	West
Month
1	5012.34	7551.55	8550.33
2	6428.75	5540.10	10864.87
3	6082.75	8863.80	14087.99
4	6420.63	7647.28	8865.57
5	8782.68	5651.30	10962.00
6	6442.85	3954.90	9020.65
7	7180.45	10155.59	10150.25
8	6031.55	7767.60	11567.37
9	70532.44	83228.39	127160.06
10	87858.60	92034.70	151780.43
11	19478.10	24048.59	33196.52
12	7625.65	6812.70	11831.54



region_sales.plot(kind='bar', ylabel='Revenue', title='Region Monthly Revenue');

Yearly changes in revenue



changes = pd.DataFrame(df.groupby([df.Date.dt.year])['Revenue'].sum())
changes

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Revenue
Date
2015	24883.84
2016	444701.72
2017	439585.31



changes.sort_values('Date').plot(kind='bar');

Top 3 products



product_revenue

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Units	Revenue
Product
Bellen	6579	168175.05
Quad	6223	194032.15
Sunbell	4500	114283.09
Carlota	4371	101272.05
Aspen	4242	96382.80
Sunshine	4229	85983.80
Doublers	3646	149041.93

Our top 3 products are Bellen, Quad and Sunbell

The most productive sales Rep in the respective years. From your analysis, give 3 recommendations you think would help the company increase revenue in the following year.



salesReps = df[['SalesRep','Year','Revenue','Units']]
salesReps = pd.DataFrame(salesReps.groupby(['Year','SalesRep'])['Revenue'].sum())
salesReps.sort_values(by=['Year','Revenue'], ascending=False)

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

		Revenue
Year	SalesRep
2017	Julie	99727.32
	Mike	96062.19
	Nabil	81079.23
	Jessica	69479.74
	Adam	49712.19
	Nicole	43524.64
2016	Mike	104590.64
	Julie	98895.58
	Nabil	74576.22
	Jessica	71469.42
	Adam	49184.21
	Nicole	45985.65
2015	Julie	5827.15
	Mike	4924.95
	Jessica	4547.12
	Adam	3819.20
	Nabil	3249.03
	Nicole	2516.39

Julie Stands out in sales revenue yearly except in 2016

Conclusion/Recommendation:

The best months for sales are September, October and November. The company should look into creating jingles during these periods to further maximize profit.
Focus the ad targeted audience on East and South Regions
Finally Bellen and Quad sell most during these periods consider getting more of them.

What Evaluation Metrics Should I use?

AC Nice — Sun, 23 Jan 2022 13:37:01 +0000

When working with imbalanced data, judging your model's performance using accuracy is like pouring diesel on a burning flame, if not careful you too might burn. Avoid accuracy! Please use F1-score, However, depending on your problem definition you call settle for high precision or recall.

Note: You can't have high recall and precision at same time. You can only have either of both.

I’d like to know what your answer is as well? 😉

Let's Save A Life with AI 🤖

AC Nice — Tue, 18 Jan 2022 12:15:05 +0000

About our dataset

Cardiovascular diseases (CVDs) are the number 1 cause of death globally, taking an estimated 17.9 million lives each year, which accounts for 31% of all deaths worlwide.

Heart failure is a common event caused by CVDs and this dataset contains 12 features that can be used to predict mortality by heart failure.

Most cardiovascular diseases can be prevented by addressing behavioural risk factors such as tobacco use, unhealthy diet and obesity, physical inactivity and harmful use of alcohol using population-wide strategies.

Data Source https://www.kaggle.com/andrewmvd/heart-failure-clinical-data

Task

Create a model for predicting mortality caused by Heart Failure.

12 clinical features for predicting death events.

Our Machine Learning WorkFlow

I feel a lot comfortable defining my workflow for solving a machine learning problem before ever starting to solve those problem as that gives me a feel of direction. However, this may be different for someother persons ✍🏻

Below are the steps we are going to take to solve this machine learning problem

Problem Definition and Data Collection
Get the data ready for use (Data Preprocessing)
- Check for missing values
- Fill missing values, if any.
- Turn categorical Features to Numerical
Feature Engineering:
Feature engineering is the process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data. A feature is a property shared by independent units on which analysis or prediction is to be done. Features are used by predictive models and influence results.
Modeling
Make Predictions
Evaluate Model Performance metrics
See if we need to improve our model
Export our trained model
Load our trained model

1. Problem Definition

People with cardiovascular disease or who are at high cardiovascular risk (due to the presence of one or more risk factors such as hypertension, diabetes, hyperlipidaemia or already established disease) need early detection and management wherein a machine learning model can be of great help.

Data Collection

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

df = pd.read_csv('Datasets/heart_failure_clinical_records_dataset.csv')

df.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	age	anaemia	creatinine_phosphokinase	diabetes	ejection_fraction	high_blood_pressure	platelets	serum_creatinine	serum_sodium	sex	smoking	time	DEATH_EVENT
0	75.0	0	582.0	0.0	20.0	1	265000	1.9	130.0	1.0	0.0	4.0	1
1	55.0	0	7861.0	0.0	38.0	0	263358.03	1.1	136.0	1.0	0.0	6.0	1
2	65.0	0	146.0	NaN	NaN	0	162000	1.3	129.0	1.0	1.0	7.0	1
3	50.0	1	111.0	0.0	20.0	0	210000	1.9	137.0	1.0	0.0	7.0	1
4	65.0	1	NaN	1.0	20.0	0	327000	2.7	116.0	0.0	0.0	8.0	1

# Let's get a summary of our data set
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 299 entries, 0 to 298
Data columns (total 13 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   age                       298 non-null    float64
 1   anaemia                   299 non-null    int64  
 2   creatinine_phosphokinase  292 non-null    float64
 3   diabetes                  294 non-null    float64
 4   ejection_fraction         294 non-null    float64
 5   high_blood_pressure       296 non-null    object 
 6   platelets                 288 non-null    object 
 7   serum_creatinine          299 non-null    float64
 8   serum_sodium              294 non-null    float64
 9   sex                       298 non-null    float64
 10  smoking                   298 non-null    float64
 11  time                      297 non-null    float64
 12  DEATH_EVENT               299 non-null    int64  
dtypes: float64(9), int64(2), object(2)
memory usage: 30.5+ KB

# Check for duplicate
df.duplicated().sum()

# Let's check how correlated each feature is to another
correlation = df.corr()
correlation

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	age	anaemia	creatinine_phosphokinase	diabetes	ejection_fraction	serum_creatinine	serum_sodium	sex	smoking	time	DEATH_EVENT
age	1.000000	0.091852	-0.088883	-0.096539	0.061344	0.155650	-0.052726	0.059484	0.018416	-0.219635	0.252176
anaemia	0.091852	1.000000	-0.184689	-0.015766	0.026149	0.054518	0.060957	-0.099176	-0.109526	-0.133430	0.066270
creatinine_phosphokinase	-0.088883	-0.184689	1.000000	-0.022312	-0.054024	-0.012482	0.057265	0.074000	0.005451	-0.025688	0.073627
diabetes	-0.096539	-0.015766	-0.022312	1.000000	-0.015566	-0.043697	-0.092627	-0.154765	-0.135563	0.028746	0.001362
ejection_fraction	0.061344	0.026149	-0.054024	-0.015566	1.000000	-0.010466	0.199457	-0.146827	-0.057282	0.028818	-0.261605
serum_creatinine	0.155650	0.054518	-0.012482	-0.043697	-0.010466	1.000000	-0.181161	0.010253	-0.026469	-0.147806	0.290386
serum_sodium	-0.052726	0.060957	0.057265	-0.092627	0.199457	-0.181161	1.000000	-0.042738	0.001440	0.057033	-0.175385
sex	0.059484	-0.099176	0.074000	-0.154765	-0.146827	0.010253	-0.042738	1.000000	0.446947	-0.005693	-0.007482
smoking	0.018416	-0.109526	0.005451	-0.135563	-0.057282	-0.026469	0.001440	0.446947	1.000000	-0.019119	-0.014233
time	-0.219635	-0.133430	-0.025688	0.028746	0.028818	-0.147806	0.057033	-0.005693	-0.019119	1.000000	-0.522918
DEATH_EVENT	0.252176	0.066270	0.073627	0.001362	-0.261605	0.290386	-0.175385	-0.007482	-0.014233	-0.522918	1.000000

df.describe()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	age	anaemia	creatinine_phosphokinase	diabetes	ejection_fraction	serum_creatinine	serum_sodium	sex	smoking	time	DEATH_EVENT
count	298.000000	299.000000	292.000000	294.000000	294.000000	299.000000	294.000000	298.000000	298.000000	297.000000	299.00000
mean	60.870248	0.431438	577.688356	0.418367	38.227891	1.391104	136.697279	0.651007	0.322148	130.208754	0.32107
std	11.898166	0.496107	972.468942	0.494132	11.852295	1.034449	4.338123	0.477454	0.468085	77.365687	0.46767
min	40.000000	0.000000	23.000000	0.000000	14.000000	0.500000	113.000000	0.000000	0.000000	4.000000	0.00000
25%	51.000000	0.000000	115.000000	0.000000	30.000000	0.900000	134.000000	0.000000	0.000000	73.000000	0.00000
50%	60.000000	0.000000	249.500000	0.000000	38.000000	1.100000	137.000000	1.000000	0.000000	115.000000	0.00000
75%	70.000000	1.000000	582.000000	1.000000	45.000000	1.400000	140.000000	1.000000	1.000000	201.000000	1.00000
max	95.000000	1.000000	7861.000000	1.000000	80.000000	9.400000	148.000000	1.000000	1.000000	285.000000	1.00000

Definition of terms from the dataframe .describe() method above

Count: Number of items of in a particular feature/Column
Mean Average number (Value gotten by dividing the sum of several quantities by their number)
Std Standard Deviation (Square root of variance (How much values differ from the mean value) )
Min Minimum value in each feature or column
Max Maximum value in each feature or column
Percentile Splitting our data into different equal segments (25, 50, 75) etc.

# Check number of columns and rows
df.shape #This shows that our dataset containes 299 rows and 13 columns

(299, 13)

# Let's checkout the column names
j = 0
for i in df.columns:
    j+=1
    print(j, i.upper())

1 AGE
2 ANAEMIA
3 CREATININE_PHOSPHOKINASE
4 DIABETES
5 EJECTION_FRACTION
6 HIGH_BLOOD_PRESSURE
7 PLATELETS
8 SERUM_CREATININE
9 SERUM_SODIUM
10 SEX
11 SMOKING
12 TIME
13 DEATH_EVENT

df['age'].T.hist(bins=50);

# Let's Visualize our correlations better
# sns.set(font_scale=1)
fig, ax = plt.subplots(figsize=(15,10))
ax = sns.heatmap(correlation, annot=True, fmt='.2f', cmap='YlGnBu', linewidths=.05);

pd.crosstab(df.age, df.DEATH_EVENT)

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

DEATH_EVENT	0	1
age
40.000	7	0
41.000	1	0
42.000	6	1
43.000	1	0
44.000	2	0
45.000	13	6
46.000	2	1
47.000	1	0
48.000	0	2
49.000	3	1
50.000	18	8
51.000	3	1
52.000	5	0
53.000	9	1
54.000	1	1
55.000	14	3
56.000	1	0
57.000	1	1
58.000	8	2
59.000	1	3
60.000	20	13
60.667	1	1
61.000	4	0
62.000	4	1
63.000	8	0
64.000	3	0
65.000	18	8
66.000	2	0
67.000	2	0
68.000	3	2
69.000	1	2
70.000	18	7
72.000	2	5
73.000	3	1
75.000	5	6
77.000	1	1
78.000	2	0
79.000	1	0
80.000	2	5
81.000	1	0
82.000	0	3
85.000	3	3
86.000	0	1
87.000	0	1
90.000	1	2
94.000	0	1
95.000	0	2

2. Data Preprocessing

Check if there are missing values

df.isna()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	age	anaemia	creatinine_phosphokinase	diabetes	ejection_fraction	high_blood_pressure	platelets	serum_creatinine	serum_sodium	sex	smoking	time	DEATH_EVENT
0	False	False	False	False	False	False	False	False	False	False	False	False	False
1	False	False	False	False	False	False	False	False	False	False	False	False	False
2	False	False	False	True	True	False	False	False	False	False	False	False	False
3	False	False	False	False	False	False	False	False	False	False	False	False	False
4	False	False	True	False	False	False	False	False	False	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...
294	False	False	False	False	False	False	True	False	False	False	False	False	False
295	False	False	False	False	False	False	False	False	False	False	False	False	False
296	False	False	False	False	False	False	True	False	False	False	False	False	False
297	False	False	False	False	False	False	True	False	False	False	False	False	False
298	False	False	False	False	False	False	True	False	False	False	False	False	False

299 rows × 13 columns

df.isna().sum()

age                          1
anaemia                      0
creatinine_phosphokinase     7
diabetes                     5
ejection_fraction            5
high_blood_pressure          3
platelets                   11
serum_creatinine             0
serum_sodium                 5
sex                          1
smoking                      1
time                         2
DEATH_EVENT                  0
dtype: int64

✍🏻 Now, with the help of pandas .isna() method we are able to find out which columns have empty values in them

Fill the missing values

When handling missing values in during your data exploration and engineering there are different ways to handle missing values.

Fill them with the mean, mode or median data of their parent column
Removing the samples with missing data
Use of Unsupervised learning approach to predict and fill the data

list(df.columns)

['age',
 'anaemia',
 'creatinine_phosphokinase',
 'diabetes',
 'ejection_fraction',
 'high_blood_pressure',
 'platelets',
 'serum_creatinine',
 'serum_sodium',
 'sex',
 'smoking',
 'time',
 'DEATH_EVENT']

Split our data into Features and Label (Independent and Dependent Variables)

# Split the data into features and labels
X = df.drop('DEATH_EVENT', axis=1)
y = df['DEATH_EVENT']

# We can use the SimpleImputer Class in scikit learn to do this but
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer


# Let's Define the columns to fill
categoricals = ['platelets', 'high_blood_pressure']
numerical = ['age',
             'anaemia',
             'creatinine_phosphokinase',
             'diabetes',
             'ejection_fraction',
             'serum_creatinine',
             'serum_sodium',
             'sex',
             'smoking',
             'time'] # Removed the two categorical feature (high_blood_pressure and platelets)

# Define the way SimpleImputer will fill the missing values
categorical_imputer = SimpleImputer(strategy='constant', fill_value='None')
numerical_imputer = SimpleImputer(strategy='mean')

# Create the Imputer 
transformer = ColumnTransformer([('categoricals', categorical_imputer, categoricals),
                                 ('numerical', numerical_imputer, numerical)])

new_X = transformer.fit_transform(X)

Congratulations!! 😎 💃
We've successfully filled and converted all our data to numbers

4. Modelling

Based on our problem and data, what machine learning model should we use? Let the audience decide 😉

Okay, you got it right 🤣, This is a classification problem because we are predicting whether our output is one thing or another i.e. whether it's 1 or 0, True or False, Rice or beans etc. In otherwords, we call this a Binary Classification

from sklearn.ensemble import RandomForestClassifier

# Instantiate our model classes
random_forest_model, ada_boost = RandomForestClassifier(), AdaBoostClassifier()

# Split our data into train and test splits
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(new_X, y, train_size=.8, random_state=21)

# Let's Checkout the shape of our data to understand how our data is being split
X_train.shape, X_test.shape, y_train.shape, y_test.shape

((239, 12), (60, 12), (239,), (60,))

Now Let's Train our machine learning model to find patterns in our data

random_forest_model.fit(X_train, y_train)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-306-7b70e76ee335> in <module>
----> 1 random_forest_model.fit(X_train, y_train)


/opt/anaconda3/lib/python3.8/site-packages/sklearn/ensemble/_forest.py in fit(self, X, y, sample_weight)
    302                 "sparse multilabel-indicator for y is not supported."
    303             )
--> 304         X, y = self._validate_data(X, y, multi_output=True,
    305                                    accept_sparse="csc", dtype=DTYPE)
    306         if sample_weight is not None:


/opt/anaconda3/lib/python3.8/site-packages/sklearn/base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    431                 y = check_array(y, **check_y_params)
    432             else:
--> 433                 X, y = check_X_y(X, y, **check_params)
    434             out = X, y
    435 


/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0


/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
    812         raise ValueError("y cannot be None")
    813 
--> 814     X = check_array(X, accept_sparse=accept_sparse,
    815                     accept_large_sparse=accept_large_sparse,
    816                     dtype=dtype, order=order, copy=copy,


/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0


/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    614                     array = array.astype(dtype, casting="unsafe", copy=False)
    615                 else:
--> 616                     array = np.asarray(array, order=order, dtype=dtype)
    617             except ComplexWarning as complex_warning:
    618                 raise ValueError("Complex data not supported\n"


/opt/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order, like)
    100         return _asarray_with_like(a, dtype=dtype, order=order, like=like)
    101 
--> 102     return array(a, dtype, copy=False, order=order)
    103 
    104 


ValueError: could not convert string to float: 'None'

Now, why are we getting an error? What did we do wrong? What can we do about it ?

# Let's checkout the unique values in our X,y set
y.unique(), new_X[::-1]

(array([1, 0]),
 array([['None', '0', 50.0, ..., 1.0, 1.0, 285.0],
        ['None', '0', 45.0, ..., 1.0, 1.0, 280.0],
        ['None', '0', 45.0, ..., 0.0, 0.0, 278.0],
        ...,
        ['162000', '0', 65.0, ..., 1.0, 1.0, 7.0],
        ['263358.03', '0', 55.0, ..., 1.0, 0.0, 6.0],
        ['265000', '1', 75.0, ..., 1.0, 0.0, 4.0]], dtype=object))

Now, because our new_X label for some reasons still contains strings rather than numbers, our machine learning model has refused to work with such data and therefore cause it to raise an error

def fix_object_data(X):
    """
        This function will help us fix the string value errors in our dataset by turning string dtypes to numeric
        X: DataFrame or dict
    """
    for label, content in X.items():
        if pd.api.types.is_string_dtype(content):
            X[label] = pd.to_numeric(X[label], errors='coerce')
    return X

Lets remind ourselves of the datatypes involved in our dataset

X.dtypes

age                         float64
anaemia                       int64
creatinine_phosphokinase    float64
diabetes                    float64
ejection_fraction           float64
high_blood_pressure          object
platelets                    object
serum_creatinine            float64
serum_sodium                float64
sex                         float64
smoking                     float64
time                        float64
dtype: object

fix_object_data(X).dtypes

age                         float64
anaemia                       int64
creatinine_phosphokinase    float64
diabetes                    float64
ejection_fraction           float64
high_blood_pressure         float64
platelets                   float64
serum_creatinine            float64
serum_sodium                float64
sex                         float64
smoking                     float64
time                        float64
dtype: object

X = fix_object_data(X)

One thing i recommend you do before fill missing values is visulizing the data frequency for outliers, so as to know what averaging method would better suit for filling of the missing values

def value_counts(data:pd.DataFrame, key='age'):
    columns = data.columns
    val_count = {}

    # Loop throught our data 
    for col in columns:
        val_count[col] = data[col].value_counts()
    return pd.DataFrame(val_count[key])

# Now let's consider filling this values one after the other since our features are not much
X['age'] = X['age'].fillna(X['age'].mean())
X['creatinine_phosphokinase'].fillna(X['creatinine_phosphokinase'].median(), inplace=True)
X['diabetes'].fillna(X['diabetes'].median(), inplace=True)
X['ejection_fraction'].fillna(X['ejection_fraction'].mean(), inplace=True)
X['high_blood_pressure'].fillna(X['high_blood_pressure'].mean(), inplace=True)
X['platelets'].fillna(X['platelets'].mean(), inplace=True)
X['serum_sodium'].fillna(X['serum_sodium'].median(), inplace=True)
X['sex'].fillna(X['sex'].median(), inplace=True)
X['smoking'].fillna(X['smoking'].median(), inplace=True)
X['time'].fillna(X['time'].median(), inplace=True)

X.isna().sum()

age                         0
anaemia                     0
creatinine_phosphokinase    0
diabetes                    0
ejection_fraction           0
high_blood_pressure         0
platelets                   0
serum_creatinine            0
serum_sodium                0
sex                         0
smoking                     0
time                        0
dtype: int64

Now we've fixed our data, lets split and fit into our model for training

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X,y, train_size=.8, random_state=21)

# Now let's train our train our model
model = RandomForestClassifier(n_estimators=120,max_depth=10)
model.fit(X_train, y_train)

RandomForestClassifier(max_depth=10, n_estimators=120)

5. Make predictions using our model

model.predict(X_test)

array([1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,
       1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0])

# Have a look at what our X_test looks like
X_test.head()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	age	anaemia	creatinine_phosphokinase	ejection_fraction	high_blood_pressure	platelets	serum_creatinine	serum_sodium	sex	smoking	time
61	50.0	0	318.0	40.0	1.0	216000.000000	2.3	131.0	0.0	0.0	60.0
297	45.0	0	2413.0	38.0	0.0	262269.408112	1.4	140.0	1.0	1.0	280.0
55	95.0	1	371.0	30.0	0.0	461000.000000	2.0	132.0	1.0	0.0	50.0
243	73.0	1	1185.0	40.0	1.0	220000.000000	0.9	141.0	0.0	0.0	213.0
95	58.0	1	133.0	60.0	1.0	219000.000000	1.0	141.0	1.0	0.0	83.0

6. Evaluate our model Performance

There are different classification evaluation metrics available evaluate our model's performance however the use of these metrics depends heavily on what problems you are solving.
They include as follows:

Accuracy : Default metric for classification problems. Not the best for imbalanced classes
Precision: Higher precision leads to less false positives
Recall: Higher recall leads to less false negatives
F1 Score: Usually a good overall metrics for classification model
Confusion Matrix: Evaluate the accuracy of a classification

from sklearn.metrics import confusion_matrix, accuracy_score, precision_score

y_pred = model.predict(X_test)
accuracy_score(y_test, y_pred)

0.9

8. Export our model

There are 2 ways we can save our model

Using the pickle module
Using the joblib module

1. Using the pickle module

import pickle as pkl

# Export with pickle
pkl.dump(model, open('your_first_model_pkl.pkl','wb')) #wb -> write binary

# Load with pickle
loaded_model_pkl = pkl.load(open('your_first_model_pkl.pkl', 'rb')) #rb -> read binary

# Make prediction
loaded_model_pkl.predict(X_test)

array([1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,
       1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0])

2. Using the Joblib module

import joblib as jbl

# Export with joblib
jbl.dump(model, 'your_first_model_pkl.joblib')

# Load with joblib
loaded_model_jbl = jbl.load('your_first_model_pkl.joblib')

# Make prediction
loaded_model_jbl.predict(X_test)

array([1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,
       1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0])

Teach me RANSAC Algorithm like I'm 5

AC Nice — Mon, 17 Jan 2022 13:51:10 +0000

I was in a computer vision class when I first heard about the RANSAC algorithm in OpenCV, at first I didn't quite get it, the instructor tried his best though, but I guess I'm a little dumb 😅. Hence my reason for writing this piece so anyone still struggling to figure it out can take deep breath of fresh air 😊.

RANSAC stands for Random Sample Consensus, an effective algorithm used for treating data affected by outliers.

RANSAC is able to treat these kinds of data by grouping the data points into 2 separate categories:

The inlier set and
The outlier set

By doing this, RANSAC eliminates your concerns of handling outliers. The outliers are simply ignored and now you only have the inlier data points to work with.

Lets have a look at the regression line fitted through the 2D data points.

You might notice that some of the data points fall in line or close to our fitting line while others fall out. The question now is, how can we separate the inliers from the outliers. You see where the RANSAC algo comes in? 😉😜

The RANSAC algorithm iteratively looks for the best fitting with the highest number of inliers. The fitting point with the highest number of inlier is choosen and that is the basics of what RANSAC is all about.

Conclusion:
Random Sample Consensus is basically a 4 ways procedure of dealing with outliers.

Sample a small subset of datapoints and treat them as inliers
Take the potential inliers and compute the model parameters
Score how many data points supports the fitting.
Repeat and take the best fitting score. 😁💃

RANSAC can be applied during image analysis to determine the position and orientation of a robot by analyzing the associated camera images.

My name is AC Nice and I love Computer Vision

Simple Face Classification Problem using Python

AC Nice — Mon, 17 Jan 2022 10:46:27 +0000

import matplotlib.pyplot as plt
import matplotlib.image as pimg 
import tensorflow as tf
import pandas as pd
import numpy as np
import random 
import os

from google.colab import drive
drive.mount('/content/drive', force_remount=True)

Mounted at /content/drive

Problem Definition

We are trying to distinguish between images that are real from images that are fake. This will enable me or any firm planning to integrate this fish out fake image upload from users on their platform.

In order to achieve this we have downloaded a dataset of faces from kaggle.com/ciplab/real-and-fake-face-detection which is categorized into fake and real classes.

Steps we will take to get this project done are:

Get the image data
Visualize and get to understand what our data looks like
Prepare our data for our neural network
Build a CNN base model for prediction
Evaluate and Improve our model
Are we satisfied? if not, REPEAT 😅

Deployment

After building this model it will be deployed to google GCP (Google Cloud Platform) where it will be rendered as an API

Get the image data

path = '/content/drive/MyDrive/real_and_fake_face/'
image_classes = os.listdir(path)

# Lets checkout how many paths are there in our image_path
image_classes

['not_face', 'face']

# Our image path contains an extra class '.ipynb_checkpoints', so let's see if we can take it out
! rm -rf .ipynb_checkpoints*
image_classes

['not_face', 'face']

# Check the amount of data we are trying to work with
for dir in os.listdir(path):
  print(f'There are at least {len(os.listdir(os.path.join(path, dir)))} {dir} images to work with.')

There are at least 960 not_face images to work with.
There are at least 1078 face images to work with.

Visualize our Images

So far we've gotten our image data, therefore in order to get a better understanding of what they look like, let's visualize them.

We will create a helper function to help us visualize what these images look like. Let's code.

def show_image(path='/content/drive/MyDrive/real_and_fake_face/', class_name=''):
  if class_name == '':
    print('Please pass a class name')
  else:
    try:
      img_path = os.listdir(os.path.join(path, class_name))
      selected_img = random.choice(img_path)
      image = plt.imread(os.path.join(path, class_name, selected_img))

      # Let's plot the image
      plt.figure(figsize=(15, 5))
      plt.imshow(image)
      plt.axis('off')
      plt.title(f'File Name: {selected_img} \n Class Name: {class_name} \n Shape: {image.shape}')

    except FileNotFoundError:
      print(f'Sorry no such class name "{str(class_name).capitalize()}" was found in path {path}')

# Let's Checkout some real images
show_image(class_name='face')

# Let's checkout some fake image
show_image(class_name='not_face')

Prepare our data for our neural network

We've visualized a fair few amount of images, now lets prepare our data, to feed to a neural network

from tensorflow.keras.preprocessing.image import ImageDataGenerator

BATCH_SIZE = 16
IMG_HEIGHT = 124
IMG_WIDTH = 124
PATH = '/content/drive/MyDrive/real_and_fake_face/'

# Instantiate ImageDataGenerator
image_gen = ImageDataGenerator(rescale=1./255, horizontal_flip=True, validation_split=0.2)

# scale the images
train_data = image_gen.flow_from_directory(directory=PATH,
                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                           class_mode='binary',
                                           subset='training',
                                           batch_size=BATCH_SIZE,
                                           seed=42,
                                           shuffle=True)

test_data = image_gen.flow_from_directory(directory=PATH,
                                          target_size=(IMG_HEIGHT,IMG_WIDTH),
                                          class_mode='binary',
                                          subset='validation',
                                          batch_size=BATCH_SIZE,
                                          seed=42,
                                          shuffle=True)

Found 1630 images belonging to 2 classes.
Found 407 images belonging to 2 classes.

image_classes = [*train_data.class_indices]
image_classes

['face', 'not_face']

Build a CNN base model for prediction

Now let's build a model to try predict what class an image falls into, let's do this!!

We took a few look at what our processed image data looks like and we found out the image not yet normalized. Remember that neural network best perform well on normalized data. So we need to normalize the data before feeding it to our neural network

# Build our model
model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
    tf.keras.layers.Conv2D(16, 2, activation='relu'),
    tf.keras.layers.MaxPool2D(), 
    tf.keras.layers.Conv2D(32, 2, activation='relu'),
    tf.keras.layers.MaxPool2D(),
    tf.keras.layers.Conv2D(64,2, activation='relu'),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])

# Compile our model
model.compile(loss=tf.keras.losses.binary_crossentropy,
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

# Fit the data to our model
history_1 = model.fit(train_data, 
                      epochs=5, 
                      validation_data=test_data, 
                      validation_steps=len(test_data), 
                      steps_per_epoch=len(train_data))

Epoch 1/5
102/102 [==============================] - 435s 4s/step - loss: 0.3324 - accuracy: 0.8472 - val_loss: 0.3306 - val_accuracy: 0.8624
Epoch 2/5
102/102 [==============================] - 20s 198ms/step - loss: 0.1399 - accuracy: 0.9466 - val_loss: 0.0866 - val_accuracy: 0.9607
Epoch 3/5
102/102 [==============================] - 20s 197ms/step - loss: 0.0904 - accuracy: 0.9699 - val_loss: 0.0980 - val_accuracy: 0.9631
Epoch 4/5
102/102 [==============================] - 20s 199ms/step - loss: 0.0696 - accuracy: 0.9779 - val_loss: 0.1108 - val_accuracy: 0.9582
Epoch 5/5
102/102 [==============================] - 19s 191ms/step - loss: 0.0553 - accuracy: 0.9822 - val_loss: 0.0677 - val_accuracy: 0.9803

Plot a history graph to visualize how our model is learning

def plot_history(history):
  """
  Returns a plots of model learning history object

  Args:
    history: Model's learning history object

  Returns:
    A plot of of model learning history
  """

  train_loss = history.history['loss']
  val_loss = history.history['val_loss']

  train_accuracy = history.history['accuracy']
  val_accuracy = history.history['val_accuracy']

  # Plot the histories

  plt.plot(train_loss)
  plt.plot(val_loss)
  plt.xlabel('Epochs')
  plt.legend(['train loss','val loss']);

plot_history(history_1)

# Create a function to enable us check our model performance on unseen data
def decode_image(image_path):
  image = tf.io.read_file(image_path)
  decoded_image = tf.io.decode_image(image,channels=3)

  # Resize the image
  image = tf.image.resize(decoded_image, size=[IMG_HEIGHT, IMG_WIDTH])

  # Normalize the image
  image = image/255.

  # Predict
  prediction = model.predict(tf.expand_dims(image, axis=0))
  print(prediction)
  print(int(tf.round(prediction)))
  prediction = image_classes[int(tf.round(prediction))]

  # plot
  plt.figure(figsize=(12, 8))
  plt.imshow(decoded_image)
  plt.title(prediction)
  plt.axis('off')

image_classes

['face', 'not_face']

decode_image('/content/drive/MyDrive/me.jpg')

[[0.00042969]]
0

decode_image('/content/drive/MyDrive/cga-ai.png')

[[0.8060113]]
1

Forem: AC Nice

How to create LinkedIn Job API using Python

Content

Intro

Requirements

What you can use LinkedIn Jobs API for

Getting Started

Install the Python libraries

Quick Usage From Python Interpreter

Job Search API

Jobs API

Sales Trend Analysis With Pandas

Assessment

Some useful Insights

Revenue by region

Revenue by sales Rep

Revenue by products

Sales trend

Plot Yearly Sales Chart/Trend

Region Monthly Revenue

Yearly changes in revenue

Top 3 products

The most productive sales Rep in the respective years. From your analysis, give 3 recommendations you think would help the company increase revenue in the following year.

Conclusion/Recommendation:

What Evaluation Metrics Should I use?

Let's Save A Life with AI 🤖

About our dataset

Task

Our Machine Learning WorkFlow

1. Problem Definition

Data Collection

Definition of terms from the dataframe .describe() method above

2. Data Preprocessing

Check if there are missing values

Fill the missing values

Split our data into Features and Label (Independent and Dependent Variables)

Congratulations!! 😎 💃 We've successfully filled and converted all our data to numbers

4. Modelling

Now Let's Train our machine learning model to find patterns in our data

Now, why are we getting an error? What did we do wrong? What can we do about it ?

Now we've fixed our data, lets split and fit into our model for training

5. Make predictions using our model

6. Evaluate our model Performance

8. Export our model

1. Using the pickle module

2. Using the Joblib module

Teach me RANSAC Algorithm like I'm 5

Simple Face Classification Problem using Python

Problem Definition

Steps we will take to get this project done are:

Deployment

Get the image data

Visualize our Images

Prepare our data for our neural network

Build a CNN base model for prediction

Plot a history graph to visualize how our model is learning

Congratulations!! 😎 💃
We've successfully filled and converted all our data to numbers