Forem: allan-pg

[Boost]

allan-pg — Sun, 08 Mar 2026 22:27:16 +0000

allan-pg

Mar 4

Designing a Modern Data Warehouse: Combining Bill Inmon and Ralph Kimball in a Hybrid Medallion Architecture

#database #datawarehouse #dataarchitecture #dataengineering

Comments

3 min read

all you need to know on data Warehouse

allan-pg — Wed, 04 Mar 2026 23:30:29 +0000

Designing a Modern Data Warehouse: Combining Bill Inmon and Ralph Kimball in a Hybrid Medallion Architecture

allan-pg ・ Mar 4

#database #datawarehouse #dataarchitecture #dataengineering

Designing a Modern Data Warehouse: Combining Bill Inmon and Ralph Kimball in a Hybrid Medallion Architecture

allan-pg — Wed, 04 Mar 2026 10:53:13 +0000

Hey devs and data enthusiasts! Ever felt like your data warehouse is a chaotic junk drawer full of potential but impossible to navigate? You're not alone. In the world of big data, choosing the right architecture can make or break your analytics game. Today, we're diving into two legendary approaches: Ralph Kimball's dimensional magic and Bill Inmon's corporate Warehouse. But here's the twist we'll combine them together into a hybrid powerhouse using modern data layers. We're going to create a blueprint for a Hybrid Inmon-Kimball architecture, where Inmon's structured approach is utilized for integration in the silver layer and Kimball's user-friendly approach is utilized for blazing query speeds in the gold layer! Let's roll!

Bill Inmon's Corporate Information Factory (The Top-Down Approach)

Imagine your data warehouse like a very big, well-organized central library. This is the idea behind Bill Inmon’s approach often called the Corporate Information Factory. First, you build an enterprise data warehouse where all data from different systems is cleaned, organized, and stored. An enterprise Data warehouse is normalised upto the 3rd normal form. Think of it like sorting all your LEGO pieces carefully before building anything. After building an enterprise data warehouse, you then create smaller sections called data marts for specific teams like finance or marketing. The advantage is that the data is very accurate, consistent, and reliable across the company. The downside is that it takes a long time to design and build at the beginning, and queries can sometimes be slower because the data is split into many related tables.

Inmon’s idea is simple: build a strong, clean foundation first, then everything else becomes easier.

Ralph Kimball's Dimensional Modeling (Bottom-Up Approach)

Kimball’s Dimensional Data Warehouse collects data from different operational systems, cleans and integrates it through ETL, and stores it in a central warehouse designed using star schemas. The warehouse keeps detailed (atomic) data but organizes it in a way that is easy for reporting and analysis. In this design, fact tables store measurable business events, usually numerical values like sales amount, quantity sold, or transaction count. However, not every number is a fact. For example, a customer’s phone number or postal code is numeric, but it is not a fact because it does not represent a measurable business event. Facts answer questions like “How much?” “How many?” “How often?”

Dimension tables contain descriptive information used to filter, group, or label the facts. Examples include customer name, product category, store location, or date. If you ask, “Total sales by product category by month,” the sales amount comes from the fact table, while product category and month come from dimension tables.

Kimball’s idea is simple: deliver useful data quickly and make it easy for people to analyze.

Hybrid Inmon-Kimball in a Medallion Architecture

A hybrid data warehouse model simply combines the strengths of both Inmon and Kimball in a modern layered setup. First, raw data lands in the Bronze layer, just as it comes from source systems. Then in the Silver layer, you apply Inmon’s approach by cleaning, integrating, and organizing the data into a structured, normalized format to create one trusted version of the truth for the whole company. This layer focuses on accuracy, consistency, and removing duplicates. Finally, in the Gold layer, you apply Kimball’s approach by transforming that clean data into easy-to-use star schemas and data marts designed for fast reporting and dashboards.

Wrapping It Up:

It doesn`t matter if you’re building a small personal project or a massive enterprise data system, the key is to start small and build in layers. Think of your data in stages that is raw data, cleaned data, and ready-to-use then pick tools that make it easy to manage, like Snowflake or Databricks. Focus on iterating and improving gradually instead of trying to do everything at once. This way, your data warehouse becomes more flexible, easier to use, and actually helpful for the people who need it.

From Messy to Masterpiece: Cleaning FIFA Data with Python ⚽🚀

allan-pg — Wed, 19 Feb 2025 06:57:38 +0000

Introduction

Data Cleaning is a very important part of data analysis. Clean Data ensures that your analysis is correct and your insights are reliable when used in decision making. It also improves quality of your data

Objectives:

Ensure that all columns are clearly named
Ensure that columns have the correct datatypes
Remove all unnecessary information from the dataset Lets Get Started:

Import necessary python libraries

import numpy as np
import pandas as pd
import warnings
warnings.filterwarnings('ignore')

import the csv data set from the folder where its saved

df = pd.read_csv(r'...\fifa21 raw data v2.csv')

set your data frame such that it displays all columns since the dataset has many columns.

pd.set_option('display.max_rows', 1000)
pd.set_option('display.max_columns', 1000)
pd.set_option('display.width', 1000)

The benefit is that not only all columns are well displayed, but also the printed rows can be larger than
the usual ~100 characters limit.

Display the first five rows

df.head()

Check how many rows and columns make up your dafaframe

df.shape
print('The number of rows are:', df.shape[0])
print('The number of columns are:', df.shape[1])

Display a list of all column names

for x in df.columns.tolist():
   print(x)

Make a copy of your data set so as to retain an original copy

df1 = df.copy()

Remove unnecessary columns

df1 = df1.drop(['Name', 'photoUrl', 'playerUrl'], axis = 1)

Rename columns

df1 = df1.rename(columns = {
"LongName": "Name",
"↓OVA":"Overall Rating(%)",
"POT": "Potential(%)",
"BOV": "Best Overall(%)",
"BP": "Best Position",
"W/F": "Weak Foot",
"SM": "Skill Moves",
"A/W": "Attacking Work Rate",
"D/W": "Defensive Work Rate",
"IR": "International Reputation",
"PAC": "Pace",
"SHO": "Shooting",
"PAS": "Passing",
"DRI": "Dribbling",
"DEF": "Defense",
"PHY": "Physicality"
})

Remove whitespaces in the club column

#Remove whitespaces from the club column
df1['Club'] = df1['Club'].str.lstrip()

Clean contract column using a function

#define a function to change contract column values
def contract_status(value):
     if 'On Loan' in value:
       value = 'On Loan'
       return value
     elif '~' in value:
       value = 'Active'
       return value
     else:
       value = 'Free'
       return value

#apply the function on contract column
df1['Contract'] = df1['Contract'].apply(contract_status).astype('category')

Rename Contract column

#Rename the contract column
df1 = df1.rename(columns = {'Contract': 'Contract Status'})

Process of cleaning contract column

A function was defined to change the row values from '2004 ~ 2021', 'On Loan' and 'Free' to 'Active', 'On Loan' and 'Free' 'Contract' column was renamed to 'Contract Status
Also, the data type was changed to category.

Check Height and Weight Columns

#Check for unique values
for column in df[['Height', 'Weight']]:
     value = df1[column].unique()
     print(f'{column}\n{value}.\n')

Create Function to convert height to cm

#Function to convert height to cm
def convert_height(value):
   if 'cm' in value:
       value = int(value[:-2])
       return value
   else:
       feet, inches = value.split("'")
       total_inches = int(feet) * 12 + int(inches[:-1])
       height_cm = total_inches * 2.54
       return round(height_cm, 2)

df1['Height'] = df1['Height'].apply(convert_height).astype('int64')
df1['Height'].unique()

Create function to convert weight to kg

def convert_weight(value):
    if 'kg' in value:
       value = value.strip('kg')
       return value
    else:
       value = value.strip('lbs')
       Weight = round((float(value) * 0.45359237), 2)
       return Weight

df1['Weight'] = df1['Weight'].apply(convert_weight).astype('int64')
df1['Weight'].unique()

Process of Cleaning 'Height' and 'Weight' columns

Create a function to convert Height
values with cm remain the same except that we extract only value without 'CM'
value in feet and inches we convert them to CM
Lastly, convert Height column to int

Process of cleaning Weight column

create a function to convert LBS to kgs
If value is in "KGS" it remains the same
If value is in "LBS" convert them to KGS by multiplying by 0.4535...

Conclusion

In this project, we embarked on a comprehensive data cleaning and transformation journey with the FIFA dataset. Our primary objective was to prepare the data for analysis by addressing inconsistencies, refining data types, and enhancing its usability.

In conclusion, data cleaning and transformation are foundational steps in any data analysis project.By addressing inconsistencies, refining data types, and enhancing data quality, we have set the stage
for more meaningful and insightful analyses. The clean and structured dataset is now well-equipped for advanced analytics, visualizations, and modeling.

Follow me on Github and Linked In for a more details on the project

📌Are you interested in being a data analyst👩‍💻 that stands out from the rest? This can only happen when you master the skills that no one talks about the ones that turn data into decisions and insights📊 into impact.

allan-pg — Fri, 07 Feb 2025 13:07:50 +0000

Beyond the Code: The Non-Technical Superpowers Every Data Analyst Needs to Succeed.

allan-pg ・ Feb 7 '25

#dataanalysis #datascience #dataengineering #softskills

Beyond the Code: The Non-Technical Superpowers Every Data Analyst Needs to Succeed.

allan-pg — Fri, 07 Feb 2025 13:01:20 +0000

Introduction

When I was starting out on data analysis, I thought it is all about beautiful dashboards, writing complex SQL Queries and Python script. While technical skills are important, non-technical skills will make you a top notch data analyst that stands out from the rest. These skills will help you deliver impactful insights from raw data, making your work even more valuable.

These are just some crucial non-technical skills every data analyst should develop and how essential they are in today's data-driven world.

1. Problem-Solving

As a data analyst you are always solving problems, from understanding how do you retain customers to why are sales declining you find solutions to problems.

How to develop skills in problem-solving

Continuous improvement of your projects constantly
Being able to break down complex project's into small and manageable tasks
Learning different methods of analyzing your projects

2. Collaborating and Working together as a team

You will often be working as a team to ensure better results and outcomes. For example you will need to work with the marketing team to ensure there is better sales. Good collaboration ensures smoother workflows and better outcomes.

How to develop skills in team work

communicate clearly with other team members
Work on projects where you work with other data analysts
Accept feedbacks and critics from others for continuous improvement

3. Attention to details

A small mistake in your data can lead to incorrect conclusions and poor decision-making. Attention to detail ensures accuracy of insights and reliability of your analysis.

How to develop skills in attention to details

Ensure data is correct by double checking the source of data and formulas used in calculations.
Use automation tools to double check your your data and the original data.

4. Ability to stay up to date

Data analysis and software development is always evolving. You should always be ready to explore new tools that helps you stay ahead of the curve.

How to stay up to date

Reading blogs and articles published online
Participating in online competitions and workshops

5. Ability to communicate effectively

As a data analyst you should be able to communicate insights to stake holders effectively. Lack of proper communication skills means you cant communicate insights effectively which makes your analysis lose value.

How to communicate effectively

Practice explaining your findings to non-technical audiences
Use visualizations to explain findings effectively

Conclusion

Non-technical skills like communication, critical thinking, and business acumen are what truly elevates your impact as a data analyst. These skills help you translate complex data into actionable insights, foster better collaboration with teams, and align your work with broader business goals.

When you are learning don't just focus on coding but invest time in improving non technical skills.

What non-technical skills have made the biggest difference in your data analytics journey? Share your thoughts in the comments below!

Mastering Data Wrangling: A Simple Guide for Developers

allan-pg — Thu, 06 Feb 2025 07:16:05 +0000

Introduction

Data wrangling is the process of turning raw data into useful data. This process involves cleaning, structuring, and enriching raw data for analysis.

What is Data Wrangling?

Data wrangling is the process of transforming and organizing raw data into a structured format. It is also known as data munging. It involves:

Data Cleaning: Removing duplicates from your dataset, handling missing values, and correcting errors.
Data Transformation: Changing formats, normalizing, and encoding data.
Data Integration: Combining data from different sources to a unified view.
Data Enrichment: Adding new relevant information to your dataset .

Why is Data Wrangling Important?

Raw data is often incomplete, inconsistent, and unstructured. Without proper wrangling, analysis can lead to incorrect conclusions.

Importance of data wrangling

Well-prepared data ensures:

Better model accuracy for machine learning.
Improved decision-making in businesses.
Enhanced data visualization and reporting.

Common Data Wrangling Techniques

Handling Missing Data

import pandas as pd

data = {'Name': ['Alice', 'Bob', None, 'David'], 'Age': [25, None, 30, 40]}
df = pd.DataFrame(data)
print(df.isnull().sum())  # Check missing values

df.fillna({'Name': 'Unknown', 'Age': df['Age'].mean()}, inplace=True)
print(df)  # Fill missing values

Removing Duplicates

df.drop_duplicates(inplace=True)

Changing Data Types

df['Age'] = df['Age'].astype(int)

Normalizing Data

df['Age'] = (df['Age'] - df['Age'].min()) / (df['Age'].max() - df['Age'].min())

Merging DataFrames

data2 = {'Name': ['Alice', 'Bob', 'David'], 'Salary': [50000, 55000, 60000]}
df2 = pd.DataFrame(data2)
merged_df = pd.merge(df, df2, on='Name', how='left')
print(merged_df)

MY GO-TO Tools for Data Wrangling

Pandas: Powerful Python library for handling structured data.
NumPy: Useful for handling numerical operations.
SQL: For structured data manipulation.

Final Thoughts

Data wrangling is an important step in any data project. Clean and structured data ensures accurate insights and better decision-making.

What’s your go-to method for data wrangling? Let me know in the comments!

Beginners guide to Machine Learning

allan-pg — Sat, 02 Nov 2024 09:33:41 +0000

Have you ever thought when you search for a product online, now you start getting recommendation's of similar products on websites like amazon? When typing something on your phone, your keyboard has word suggestions of the word you are about to type? All this has been made possible by the application of Machine Learning.

Machine Learning is a way of teaching computers to make predictions without having to program it for every single task.
Machine Learning is a subset of Artificial intelligence(AI)

Your friend started out in car sales and he has made thousands of dollars. As a data scientist he has offered you to be his business partner. He supplies the money and you will make models to predict how much the cars are to be sold. As a data scientist you ask your friend which criteria they used to predict price of a car and you identify they used prices from the past to predict the price of a car.

Machine Learning allows you to predict prices in the same way but in a more accurate manner. We will be using Decision Trees for a clear explanation because they are easier to understand. Your friend was using the below decision tree model to predict future prices.

This model only divides cars in two categories. The biggest disadvantage of this model is does not capture most factors affecting the price of cars. For example age of the car, make and model, mileage, accident history if any, the condition of the car and etc. You can use a more deeper tree to include more factors.

The above model has now included more factors first is the car manual or automatic and what is its engine capacity. These factors are refereed to as features in your model and the price we are predicting is the prediction target

Steps to building and using a Machine learning model

Data Collection: You will need to gather data you will need to build your model. For example pictures, past data on car sales for our case.
Define: Find out What type of model bests suites what you need to achieve.
Fit: Train the model by showing it your data and letting it learn.
Predict: The model makes predictions, and you tell it whether it's right or wrong. Over time, it gets better.
Evaluate: Determine how accurate the model's predictions are.

Applications of Machine Learning

Recommendations: Based on your recent search history for example on YouTube, You will have videos being recommended to you based on your search history.
Spam Filtering of messages: On your message app you have a spam folder that contains all messages that have been filtered out as spams.
Detecting fraudulent activities: Banks are now using machine learning to help curb tax evasions and detect fraudulent transactions. If your bank account transacts an average of $3,000 per month then if it starts transacting over $50,000 then that raises an eye brow.
Transport: A more practical example google map can help you choose the best alternative route based on traffic jam, distance which can help reduce the time you will take and the amount of fuel needed.

Conclusion

Machine Learning is used to interpret patterns in your data for a better decision making and prediction. Machine Learning allows you to feed your algorithm large amounts of data and it analyzes the data for better decision making. Instead of humans writing code for every action, we show the computer lots of examples, and it learns from those.

Mastering SQL Optimization: A Beginner’s Guide to Faster and More Efficient Queries

allan-pg — Sun, 27 Oct 2024 07:32:21 +0000

Introduction

Has it ever crossed your mind why some SQL queries run in a flash while others take forever?

I recently came across SQL Queries that yes, they retrieve the required data but the query have longer wait times before they execute. As a database admin it is your duty to ensure SQL Queries takes as less time as possible to execute and they are efficient. This in turn ensures faster retrieval of data while using less system resources.

What is SQL Optimization and why is it useful

SQL optimization is the process writing SQL queries that retrieves data quickly and efficiently. In large databases, unoptimized SQL queries often take long to execute and in return they use a lot of system resources. SQL optimization helps solve this problem by ensuring less execution time and less usage of system resources.

Techniques to ensure your SQL queries are optimized

Select only the columns you need

When writing SQL query ensure you select only the columns you need in your output instead of selecting all columns

Example:
Instead of

SELECT *
FROM table1

use

SELECT col1, col2, col3
FROM table1

Use LIMIT clause

LIMIT clause to restrict your SQL to return only the necessary rows you need instead of all rows

Example:
You need to get the top 10 most paid employees

Instead of

SELECT name, age, salary
FROM employees
ORDER BY salary DESC;

Do this

SELECT name, age, salary
FROM employees
ORDER BY salary DESC
LIMIT 10;

Use indexes

Use indexes in your database especially on tables you will use SELECT, JOIN, WHERE, ORDER BY. Although, avoid indexing every column, as this can increase storage and slow down INSERT, UPDATE, and DELETE
operations.

To create an index:

CREATE INDEX idx_customer_id 
ON orders (customer_id);

Avoid using functions on indexed columns

Using functions on indexed columns in the WHERE clause will slow down the query since it makes the database not use the index. Instead make sure your data in your columns are stored in the format you may need them stored. For example:
Instead of

SELECT id, name, age
FROM employee
WHERE LOWER(f_name) = 'james';

Ensure first name in the employee table is stored in lower case to avoid using a function in the indexed column f_name

Optimize JOINS in SQL

Another method of optimizing your SQL queries is by ensuring you Optimize join queries in SQL to get the fastest result possible.

Use indexes: You should create indexes on columns used in join conditions
Use INNER JOIN instead of other types of join: INNER JOIN is generally faster than OUTER JOIN because it only returns rows where there is a match in both tables.

Use appropriate Data Types

Ensure you use the correct data types for columns, and avoid using large data types for columns that has smaller data. For example, In the first name column don`t use VARCHAR(255) which is the maximum since nobody has a first name with over 200 characters instead use VARCHAR(50)

Example of Query Optimization

Instead of

SELECT *
FROM employee
LEFT JOIN  department ON employee.department_id =  department.department_id
WHERE UPPER(employee.last_name) = 'SMITH'
ORDER BY employeeage DESC;

Optimizing the SQL Query above

Specify the columns you need instead of selecting all columns
Avoid using LEFT Join since no employee that can be without a department instead use INNER JOIN
Avoid functions on indexed column i.e UPPER on last_name
instead index customer_id `

Conclusion

By following these best practices in large databases, SQL optimization can greatly improve performance of your application

Beginners guide to master web scrapping in python

allan-pg — Thu, 24 Oct 2024 08:25:58 +0000

Introduction

Recently I dived into scrapping data from YouTube by use of YouTube API to extract valuable information from various podcasts channels in Kenya. What I discovered is that there is a lot of valuable information on the web that can be used to get valuable insights. Ipon making this discovery I resorted to learn what is data scraping and how can we extract data from the web about a product or lets say anything you can think of as a data analyst.

What is web scrapping?

Web Scrapping is the process of extracting valuable information from various web pages. This information can be in form of text, pictures or links found on web pages. Web scrapping is used to perform price monitoring, price intelligence, news monitoring, lead generation, and market research.

Necessary tools and libraries needed to scrape data from the web

The following tools and libraries are essential to scrape data from the web:

Python

Python is a high level programming language used to scrape data from the web since it contains powerful libraries.

Requests

Requests is a python library used to make HTTP request to a specific URL and returns the response. To install requests in Jupyter notebook:

pip install requests

Example of making a request to URL and getting response

import requests

url = 'https://en.wikipedia.org/wiki/Motion_Picture_Association_film_rating_system'

response = requests.get(url)
print(response.content)

BeautifulSoup

BeautifulSoup makes it easy to parse HTML and XML documents and extract data from them. To install beautifulsoup on jupyter notebook :-

pip install beautifulsoup4

A simple python program that extracts data using beautifulsoup

# Import requests to send a request to the url
# Import beautifulsoup to extract data from html documents
import requests
from bs4 import BeautifulSoup


# Making a GET request
url = 'https://en.wikipedia.org/wiki/Motion_Picture_Association_film_rating_system'

response = requests.get(url)

# Parsing the HTML
soup = BeautifulSoup(response.content, 'html.parser')

# find paragraphs in the class_name of your choice 
paragraphs = soup.find_all('p', class_= 'Class name from html code')

## Loop through each paragraph and print the text
for paragraph in paragraphs:
    print(paragraph.text))

Scrapy

Scrapy is free and open source web crawling framework written in python. Unlike beautifulsoup which is used in parsing html, Scrapy handles everything from requests and parsing to data storage and handling crawling rules. Scrapy allows developers to efficiently crawl web pages and extract the desired information.
To install scrapy in your jupyter notebook

pip install scrapy

Steps to Follow during web scrapping

Identify the website to extract data from and ensure it has the data you need to perform data analysis
Inspect the web page - This is done by first opening the web page, then right click and choose inspect to inspect the HTML structure of the web page and locate the data you will need.
Make a HTTP request to the web page using the request library.
Using beautiful soup parse HTML content to find the necessary data you shall need
Once you have located the data, extract it and store it in a suitable format such as CSV or JSON

Web scrapping best practices

Avoid making too many requests at the same time as this may overload the server making the website slow for other users
Write reusable functions to make your code more readable and easy to maintain
Ensure you handle and manage errors efficiently such as handling missing data
When making requests, include a User-Agent header to mimic a regular browser and avoid being blocked.

Conclusion

Web scraping can be used to extract a lot of information from the web. Ensure to scrape data ethically and responsibly by following the outlined practices above. Happy Scraping!

Check out my recent project where I used YouTube API to extract data on podcasts channels in Kenya on Linked IN

lets connect Linked IN

5 Essential Excel Features Every Data Analyst Should Master

allan-pg — Fri, 18 Oct 2024 06:47:02 +0000

Introduction

Microsoft excel is a very powerful tool when it comes to organizing, manipulating, analyzing and visualization of data. From large businesses to small owned business, Excel remains a go to tool that handles simple calculations to advanced data modelling. Here are five key features of excel every data analyst should know and use

Pivot Tables

Pivot tables are used to summarize large datasets to derive meaningful information from them. As a data analyst, pivot tables will help you extract insights and identify trends easily without the need of writing complex formulas.

Why are Pivot Tables important?

Using pivot tables, one can extract insights without the need to write complex formulas
Pivot tables helps to save time since analyze and organize large amounts of data quickly and efficiently.
Pivot tables when combined with pivot charts makes it easier to create interactive dashboards that can be used to identify trends.

Conditional Formatting

Conditional formatting is used to highlight cells in excel based on the specified criteria. You can apply colors, icons, or data bars based on conditions, making it easier to interpret results quickly.

Why is conditional formatting a useful feature in excel?

Conditional formatting makes data more visible by use of colors, data bars and icons thus making it quick to locate data
By using alerts that flag values meeting certain criteria, a data analyst can be able to prevent issues such as missed deadlines
Helps in decision-making by drawing attention to trends, outliers, and errors.
Conditional formatting can highlight duplicates making it easier for data cleaning process

Power Query

Using power query, one cam import data from different sources for example, databases so as to clean and transform data from these sources without the need of hard coding.

In what ways does Power Query enhance the data analysis process?

Handles large datasets and supports importation of data from multiple sources
Power query automates repetitive tasks in data cleaning making it an error free process
Power query allows a data analyst to perform complex data transformation via a user friendly interface that does not require writing complex queries.

Data Validation

Data validation helps to restrict the type of data to be inputted in a cell, print an error message in case the data entered is incorrect and create drop down list to choose from.

What are the key benefits of data validation?

Ensures data integrity because it helps prevent incorrect data from being inputted in your data thus reducing time that would have been used to clean the data.
Ensures that data entered follows a consistent format. For example date formats can be set to (YYYY-MM-dd)

Look up Functions

Lookup functions help you search for data in one table and return matching results from another. Look up functions include:**

h-lookup
x-lookup
Indexmatch
v-lookup**

What makes Look Up functions a critical feature in data analysis?

It helps in faster retrieval of data needed since by use of look up functions one can located data needed quickly.
Lookup functions enable you to join or merge data from different tables or sheets by matching related columns (like IDs or product codes).

Conclusion

mastering these five Excel features ensures a data analyst can work efficiently and effectively with large data sets. By leveraging these tools, data analysts can derive meaningful information and help businesses in better decision making.

Handling Outliers 101: Why the IQR Method is Your Go-To Tool

allan-pg — Thu, 10 Oct 2024 20:18:00 +0000

Introduction

Before uncovering any insights from real-world data, it is important to scrutinize your data to ensure that data is consistent and free from errors. However, Data can contain errors and some values may appear to differ from other values and these values are known as outliers. Outliers negatively impact data analysis leading to wrong insights which lead to poor decision making by stake holders. Therefore, dealing with outliers is a critical step in the data preprocessing stage in data science. In this article, we will asses IQR method of handling outliers.

Outliers

Outliers are data points that differ significantly from the majority of the data points in a dataset. They are values that fall outside the expected or usual range of values for a particular variable. outliers occur due to various reason for example, error during data entry, sampling errors. In machine learning outliers can cause your models to make incorrect predictions thus causing inaccurate predictions.

Detecting outliers in a dataset using Jupyter notebook

Import python libraries

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
plt.style.use('ggplot')

Load your csv file using pandas

df_house_price = pd.read_csv(r'C:\Users\Admin\Desktop\csv files\housePrice.csv')

Check the first five rows of house prices data set to have a glimpse of your datafrane

df_house_price.head()

Check for outliers in the price column by use of a box plot

sns.boxplot(df_house_price['Price'])
plt.title('Box plot showing outliers in prices')
plt.show()

From the box plot visualization the price column has outlier values
Now we have to come up with ways to handle these outlier values to ensure better decision making and ensure machine learning models make the correct prediction

IQR Method of handling outlier values

IQR method means interquartile range measures the spread of the middle half of your data. It is the range for the middle 50% of your sample.

Steps for removing outliers using interquartile range

Calculate the first quartile (Q1) which is 25% of the data and the third quartile (Q3) which is 75% of the data.

Q1 = df_house_price['Price'].quantile(0.25)
Q3 = df_house_price['Price'].quantile(0.75)

compute the interquartile range

IQR = Q3 - Q1

Determine the outlier boundaries.

lower_bound = Q1 - 1.5 * IQR

Lower bound means any value below -5454375000.0 is an outlier

upper_bound = Q3 + 1.5 * IQR

Upper bound means any value above 12872625000.0 is an outlier
Remove outlier values in the price column

filt = (df_house_price['Price'] >= lower_bound) & (df_house_price['Price'] <= upper_bound)

df = df_house_price[filt]
df.head()

Box plot After removing outliers

sns.boxplot(df['Price'])
plt.title('Box plot after removing outliers')
plt.show()

Different methods of handling outlier values

Z-Score method
Percentile Capping (Winsorizing)
Trimming (Truncation)
Imputation
Clustering-Based Methods e.g DBSCAN

Conclusion

IQR method is simple and robust to outliers and does not depend on the normality assumption. The disadvantage is that it can only handle univariate data, and that it can remove valid data points if the data is skewed or has heavy tails.

Thank you
follow me on linked in and on github for more.