Forem: Hussein Mahdi

Mastering Pandas — Part 4: Data Visualization with Matplotlib & Seaborn

Hussein Mahdi — Thu, 02 Apr 2026 06:05:00 +0000

Pandas for Data Science Series — Article #4

From Clean Data to Clear Insight

In Part 3, you learned how to clean messy data and combine multiple sources into one unified DataFrame. Now that your data is ready, the next step is to communicate what's inside it — and nothing communicates data faster or more clearly than a chart.

This article covers the two most important visualization libraries in Python: Matplotlib, the core engine that powers all plotting, and Seaborn, a higher-level library built on top of it that produces beautiful statistical charts with minimal code. By the end, you'll know which tool to reach for depending on what you need to show.

We'll use this sample dataset throughout:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = {
    'Country':         ['China', 'India', 'USA', 'Brazil', 'UK'],
    'Continent':       ['Asia', 'Asia', 'North America', 'South America', 'Europe'],
    '2022 Population': [1412000000, 1380000000, 331000000, 214000000, 67000000],
    'Area (km2)':      [9597000, 3287000, 9834000, 8516000, 243000]
}
df = pd.DataFrame(data)

Part 1 — Matplotlib: The Engine

What is Matplotlib?

Matplotlib is the foundational plotting library in Python. Every chart you create in Python — whether through Pandas, Seaborn, or directly — eventually goes through Matplotlib to render the final output. Think of it as the engine: it handles the low-level work of drawing lines, shapes, colors, axes, and text.

The submodule you'll use in practice is matplotlib.pyplot, which gives you a clean, high-level interface without managing every visual detail manually:

import matplotlib.pyplot as plt

Matplotlib works across three styles, each suited to a different situation:

Style	When to use
`df.plot(kind="bar", y="col")`	Your data is already in a DataFrame — quickest way
`plt.plot(xs, ys)`	Working with raw lists, arrays, or math functions
`fig, ax = plt.subplots()`	Multiple subplots or full layout control

All three ultimately use Matplotlib to render the chart. plt.show() is always called at the end to display it.

Global Settings with `plt.rcParams`

plt.rcParams is a global settings dictionary. Any value you set here applies to every chart created afterward in the session — so you set it once at the top of your notebook and never repeat it.

plt.rcParams['figure.figsize'] = (12, 5)   # default width and height
plt.rcParams['font.size']      = 13         # default font size
plt.rcParams['figure.dpi']     = 100        # image sharpness
plt.rcParams['lines.linewidth'] = 2         # default line thickness
plt.rcParams['axes.grid']      = True       # show grid on all charts

Setting a Visual Theme with `plt.style.use()`

One line changes the entire look of your charts — colors, background, grid style, and fonts. Call it before your plot, not after.

plt.style.use("ggplot")
df.plot(kind="bar", y="2022 Population")
plt.show()

Style	Look
`"ggplot"`	Gray background, colored lines — popular R-like style
`"seaborn-v0_8-whitegrid"`	Clean, modern, white with grid
`"fivethirtyeight"`	Bold, thick lines — news article style
`"dark_background"`	Black background
`"bmh"`	Soft colors, Bayesian style
`"grayscale"`	Shades of gray only
`"tableau-colorblind10"`	Colorblind-friendly palette

Run print(plt.style.available) to see every available option.

Plotting from a DataFrame with `.plot()`

Since Pandas is built on Matplotlib, every DataFrame and Series has a .plot() method that creates a chart in one line.

df.plot(kind="bar", x="Country", y="2022 Population", figsize=(10, 5), title="Population by Country")
plt.show()

The kind= parameter selects the chart type. All common types are supported:

`kind=`	Chart
`"bar"`	Vertical bars
`"barh"`	Horizontal bars
`"line"`	Line chart (default)
`"pie"`	Pie chart
`"hist"`	Histogram
`"scatter"`	Scatter plot
`"box"`	Box plot

You can also call the chart type directly as a method — both styles produce identical output:

# These two are exactly the same:
df.plot(kind="bar", x="Country", y="2022 Population")
df.plot.bar(x="Country", y="2022 Population")

The direct method style (df.plot.bar()) is shorter and more common in practice.

Key parameters you'll use on almost every chart:

Parameter	What it does	Example
`kind`	Chart type	`"bar"`, `"line"`, `"scatter"`
`x` / `y`	Columns for axes	`x="Country"`, `y="2022 Population"`
`figsize`	Width and height in inches	`figsize=(10, 5)`
`title`	Chart title	`title="Population by Country"`
`color`	Bar/line color	`color="steelblue"`
`legend`	Show or hide legend	`legend=True`
`xlabel` / `ylabel`	Axis labels	`xlabel="Country"`
`rot`	Rotation of tick labels	`rot=45`
`grid`	Show background grid	`grid=True`
`alpha`	Transparency	`alpha=0.8`
`bins`	Bin count (histogram only)	`bins=20`

df.plot(
    kind="bar",
    x="Country",
    y="2022 Population",
    figsize=(10, 5),
    title="2022 World Population",
    color="steelblue",
    xlabel="Country",
    ylabel="Population",
    rot=0,
    grid=True,
    alpha=0.85
)
plt.show()

Plotting Directly with `plt.plot()`

When your data is not in a DataFrame — raw lists, NumPy arrays, or a mathematical function — use plt.plot() directly.

import numpy as np

xs = np.arange(-5, 5, 0.25)
ys = xs ** 2

plt.figure(figsize=(8, 5))
plt.plot(xs, ys, color="steelblue", linewidth=2, linestyle="--")
plt.title("Quadratic Function")
plt.xlabel("X")
plt.ylabel("Y")
plt.grid(True)
plt.show()

Matplotlib doesn't know your data is a curve — it connects the dots you give it with straight lines. When points are closely spaced (as with np.arange step 0.25), hundreds of short segments visually blend into a smooth curve.

Customizing plt.plot():

# Color
plt.plot(xs, ys, color='red')

# Line style
plt.plot(xs, ys, linestyle='--')    # dashed
plt.plot(xs, ys, linestyle=':')     # dotted
plt.plot(xs, ys, linestyle='-.')    # dash-dot

# Markers at each point
plt.plot(xs, ys, marker='o')        # circles
plt.plot(xs, ys, marker='s')        # squares
plt.plot(xs, ys, marker='^')        # triangles

# Everything combined
plt.plot(xs, ys, color='green', linestyle='--', linewidth=2, marker='o')
plt.show()

If you want dots without a line, use plt.scatter():

plt.scatter(xs, ys)
plt.show()

Decorating Any Chart

These functions work on any chart — whether drawn through Pandas or directly:

plt.title("Chart Title")
plt.xlabel("X Axis Label")
plt.ylabel("Y Axis Label")
plt.grid(True)
plt.legend()      # shows labels defined with label= on each line
plt.show()

When plotting multiple lines, use label= with each plt.plot() call, then plt.legend() to display them:

ys1 = xs ** 2
ys2 = xs ** 3

plt.plot(xs, ys1, label="f(x) = x²")
plt.plot(xs, ys2, label="f(x) = x³", linestyle="--")
plt.legend()
plt.show()

Multiple Subplots with Object-Oriented Style

When you need more than one chart in the same figure, use the object-oriented style. It gives you full control over each subplot independently.

# 1 row, 2 columns — side by side
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.bar(df["Country"], df["2022 Population"], color="steelblue")
ax1.set_title("Population by Country")
ax1.set_xlabel("Country")
ax1.set_ylabel("Population")

ax2.scatter(df["Area (km2)"], df["2022 Population"], color="tomato")
ax2.set_title("Area vs Population")
ax2.set_xlabel("Area (km²)")
ax2.set_ylabel("Population")

plt.tight_layout()
plt.show()

fig is the whole figure container. ax1 and ax2 are the individual plotting areas. The pattern is the same as plt.* calls, just moved to ax.set_*() methods.

Part 2 — Seaborn: Statistical Visualization

What is Seaborn?

Seaborn is a library built on top of Matplotlib, designed for statistical data visualization. It produces polished, publication-quality charts with far less code than raw Matplotlib, and works natively with Pandas DataFrames.

import seaborn as sns

Every Seaborn chart follows the same basic structure:

sns.chart_type(data=df, x='column_name', y='column_name')
plt.show()

The hue= parameter is Seaborn's most powerful feature — it automatically splits and colors your data by a category column, adding a legend with no extra work:

sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

Global Styling with `set_theme()`

Call this once at the top. Every chart afterward follows these settings automatically.

sns.set_theme(style='whitegrid', palette='deep', font_scale=1.2)

style controls the background:

sns.set_theme(style='whitegrid')   # white background with grid — most common
sns.set_theme(style='darkgrid')    # dark background with grid
sns.set_theme(style='white')       # white background, no grid
sns.set_theme(style='ticks')       # minimal, axis ticks only

palette controls the colors of chart elements:

sns.set_theme(palette='deep')        # default, rich colors
sns.set_theme(palette='muted')       # softer, less saturated
sns.set_theme(palette='pastel')      # light pastel colors
sns.set_theme(palette='colorblind')  # accessible for colorblind readers

font_scale scales all text up or down:

sns.set_theme(font_scale=1.5)    # larger text
sns.set_theme(font_scale=0.8)    # smaller text

To apply a style to one chart only without changing global settings:

with sns.axes_style('whitegrid'):
    sns.barplot(data=df, x='Continent', y='2022 Population')
    plt.show()

To reset everything back to Matplotlib defaults: sns.reset_defaults()

Relational Charts — Showing Relationships Between Numbers

Use these when you want to see how two numeric columns relate to each other.

scatterplot() — each row becomes one dot on the chart:

# Basic
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population')

# Colored by continent
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.title("Area vs Population")
plt.show()

lineplot() — connects points with a line. Best for time series or ordered data:

sns.lineplot(data=df, x='Country', y='2022 Population', hue='Continent')
plt.show()

relplot() — a single figure-level function for both scatter and line. Use kind= to switch:

sns.relplot(data=df, x='Area (km2)', y='2022 Population', kind='scatter', hue='Continent')
sns.relplot(data=df, x='Country', y='2022 Population', kind='line')
plt.show()

Distribution Charts — Showing How Values Are Spread

Use these to understand the shape and spread of a numeric column.

histplot() — bars showing how many values fall within each range:

# Basic histogram
sns.histplot(data=df, x='2022 Population')

# With smooth density curve on top
sns.histplot(data=df, x='2022 Population', kde=True)

# Split by category
sns.histplot(data=df, x='2022 Population', hue='Continent')
plt.show()

kdeplot() — a smooth density curve, more refined than a histogram:

sns.kdeplot(data=df, x='2022 Population', hue='Continent')
plt.show()

For a 2D density plot showing where two numeric columns overlap:

sns.kdeplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

rugplot() — adds small tick marks along an axis showing where individual data points are. Typically layered on top of a kdeplot:

sns.kdeplot(data=df, x='2022 Population')
sns.rugplot(data=df, x='2022 Population')
plt.show()

ecdfplot() — shows the cumulative distribution: what percentage of values fall below each point:

sns.ecdfplot(data=df, x='2022 Population')
plt.show()

displot() — one function to rule all distribution charts. Use kind= to switch:

sns.displot(data=df, x='2022 Population', kind='hist', hue='Continent')
sns.displot(data=df, x='2022 Population', kind='kde')
sns.displot(data=df, x='2022 Population', kind='ecdf')
plt.show()

Categorical Charts — Comparing Groups

Use these when one axis is a category and the other is numeric.

barplot() — shows the mean value per category, with confidence interval bars:

sns.barplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.title("Average Population by Continent")
plt.show()

Note: barplot() shows the mean, not the raw values. If you want raw counts by category, use countplot().

countplot() — counts how many rows belong to each category:

sns.countplot(data=df, x='Continent')
plt.show()

boxplot() — shows median, quartiles, and outliers. A fast way to spot unusual values:

sns.boxplot(data=df, x='Continent', y='2022 Population')
plt.show()

violinplot() — like a boxplot, but also shows the full distribution shape on both sides. More informative for larger datasets:

sns.violinplot(data=df, x='Continent', y='2022 Population')
plt.show()

stripplot() — shows every individual data point as a dot per category. Useful when you want to see actual values rather than a summary:

sns.stripplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

swarmplot() — same as stripplot but repositions overlapping points so each one is visible:

sns.swarmplot(data=df, x='Continent', y='2022 Population')
plt.show()

pointplot() — shows the mean per category as a dot, connected by lines across categories. Good for spotting trends:

sns.pointplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

catplot() — one function for all categorical charts. Use kind= to switch between them:

sns.catplot(data=df, x='Continent', y='2022 Population', kind='bar')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='box')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='violin')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='strip')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='swarm')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='point')
plt.show()

Matrix & Regression Charts — Correlations and Trends

corr() + heatmap() — the most common combination for exploring how numeric columns relate to each other:

corr = df.select_dtypes(include='number').corr()
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.title("Correlation Matrix")
plt.show()

The values range from -1.0 (perfect negative relationship) to 1.0 (perfect positive relationship). annot=True displays the number inside each cell. cmap='coolwarm' colors high values red and low values blue.

clustermap() — same as heatmap but automatically reorders rows and columns by similarity using clustering:

sns.clustermap(corr, annot=True, cmap='coolwarm')
plt.show()

pairplot() — creates scatter plots for every combination of numeric columns at once, with distribution plots along the diagonal. The fastest way to get a complete overview of your data:

# All numeric columns
sns.pairplot(df.select_dtypes(include='number'))

# Colored by a category
sns.pairplot(df, hue='Continent')
plt.show()

jointplot() — shows a scatter plot of two columns, plus the distribution of each column on the margins:

sns.jointplot(data=df, x='Area (km2)', y='2022 Population')

# With a regression line
sns.jointplot(data=df, x='Area (km2)', y='2022 Population', kind='reg')
plt.show()

lmplot() — scatter plot with a regression line fitted through the data. Supports separate lines per group:

sns.lmplot(data=df, x='Area (km2)', y='2022 Population')

# Separate regression line per continent
sns.lmplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

regplot() — the axes-level version of lmplot(). Does the same thing but doesn't support grouping. Use when you want to embed a regression plot inside a larger figure with subplots:

sns.regplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

residplot() — shows the residuals of a regression: how far each actual point is from the predicted line. A flat, random scatter around zero means the regression is a good fit:

sns.residplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

Combining Seaborn with Matplotlib

Seaborn creates the chart. Matplotlib customizes it. Both work together in the same block — you never have to choose one or the other:

sns.boxplot(data=df, x='Continent', y='2022 Population', palette='pastel')

plt.title('Population Distribution by Continent')
plt.xlabel('Continent')
plt.ylabel('Population')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

This is the standard workflow: Seaborn for the chart, Matplotlib for the title, labels, rotation, and layout.

Choosing the Right Tool

Use Matplotlib when:

Your data is in raw lists or NumPy arrays, not a DataFrame
You need multiple subplots in a specific layout
You want pixel-level control over every visual element

Use Seaborn when:

Your data is in a DataFrame and ready to visualize
You want a beautiful chart quickly with minimal code
You're doing statistical analysis — distributions, correlations, regressions, group comparisons

In most real projects, you'll use both: Seaborn to draw the chart, Matplotlib to label and polish it.

Complete Summary Table

Function	Library	Category	What it shows
`plt.rcParams`	Matplotlib	Settings	Global defaults for all charts
`plt.style.use()`	Matplotlib	Settings	Visual theme for all charts
`df.plot()`	Pandas/Matplotlib	General	Any chart type from a DataFrame
`plt.plot()`	Matplotlib	Line/Scatter	Lines and points from raw data
`plt.scatter()`	Matplotlib	Scatter	Dots from raw data
`plt.bar()`	Matplotlib	Bar	Bars from raw data
`plt.hist()`	Matplotlib	Distribution	Histogram from raw data
`plt.subplots()`	Matplotlib	Layout	Multiple charts in one figure
`sns.set_theme()`	Seaborn	Settings	Global style for all Seaborn charts
`sns.scatterplot()`	Seaborn	Relational	Relationship between two numeric columns
`sns.lineplot()`	Seaborn	Relational	Trend or time series
`sns.relplot()`	Seaborn	Relational	Wrapper for scatter and line
`sns.histplot()`	Seaborn	Distribution	Histogram with optional KDE
`sns.kdeplot()`	Seaborn	Distribution	Smooth density curve
`sns.ecdfplot()`	Seaborn	Distribution	Cumulative distribution
`sns.rugplot()`	Seaborn	Distribution	Data point ticks on axis
`sns.displot()`	Seaborn	Distribution	Wrapper for all distribution charts
`sns.barplot()`	Seaborn	Categorical	Mean per category
`sns.countplot()`	Seaborn	Categorical	Row count per category
`sns.boxplot()`	Seaborn	Categorical	Median, quartiles, outliers
`sns.violinplot()`	Seaborn	Categorical	Distribution shape per category
`sns.stripplot()`	Seaborn	Categorical	Individual data points per category
`sns.swarmplot()`	Seaborn	Categorical	Non-overlapping dots per category
`sns.pointplot()`	Seaborn	Categorical	Mean per category with trend line
`sns.catplot()`	Seaborn	Categorical	Wrapper for all categorical charts
`sns.heatmap()`	Seaborn	Matrix	Color-coded correlation matrix
`sns.clustermap()`	Seaborn	Matrix	Clustered heatmap
`sns.pairplot()`	Seaborn	Matrix	All column pairs at once
`sns.jointplot()`	Seaborn	Regression	Scatter + marginal distributions
`sns.lmplot()`	Seaborn	Regression	Scatter with regression line (supports grouping)
`sns.regplot()`	Seaborn	Regression	Scatter with regression line (no grouping)
`sns.residplot()`	Seaborn	Regression	Regression residuals

This is Part 4 of the Pandas for Data Science series. This is the last article in this series; we'll see you in another series 👨‍💻.

References

GitHub Repo: https://github.com/Hu8MA/Mastering-Pandas-Reference
Matplotlib Documentation: https://matplotlib.org/
Course : https://youtu.be/Mdq1WWSdUtw
Seaborn Documentation: https://seaborn.pydata.org/
Pandas Visualization Guide: https://pandas.pydata.org/docs/user_guide/visualization.html

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Hussein Mahdi — Wed, 18 Mar 2026 07:46:00 +0000

Pandas for Data Science Series — Article #3

Real Data Is Never Clean

In Part 2, you learned how to group and slice a DataFrame with precision. But in real projects, the data you receive rarely arrives ready to analyze. Column names have spaces. Ages are negative. Phone numbers mix letters with digits. Duplicate rows sneak in. And your data often lives across multiple files that need to be combined before you can do anything useful.

This article covers the two skills that bridge raw data and real analysis:

Data Cleaning — detecting and fixing everything wrong with your data before it corrupts your results
Merging & Joining — combining separate DataFrames into a single, unified dataset

These topics flow naturally into each other. You clean each source first, then you combine them. That's the order every data analyst follows in practice — and it's the order we'll follow here.

We'll use this sample dataset throughout the cleaning section:

import pandas as pd
import numpy as np

data = {
    "Name":             ["Alice", "bob", "Carol", "Alice", "  dave  ", "Eve"],
    "Age":              [25, -5, 200, 25, 30, None],
    "Phone":            ["055-123-4567", None, "55123x", "055-123-4567", "055-987-0000", "055abc"],
    "Email":            ["alice@mail.com", "bob@mail.com", None, "alice@mail.com", "dave@mail.com", "not-an-email"],
    "Paying_Customer":  ["Y", "N", "Yes", "Y", "N", "y"],
    "Score":            [85, 110, 72, 85, -10, 90],
}
df = pd.DataFrame(data)

Step 1 — Inspect Before You Touch Anything

The most important rule in data cleaning: look before you act. Jumping straight into fixes without understanding the full picture leads to mistakes you won't catch until much later.

These are the inspection tools you should run on every new dataset:

df.shape          # (rows, columns) — how big is this?
df.info()         # column names, data types, non-null counts
df.head()         # first 5 rows
df.tail()         # last 5 rows
df.describe()     # min, max, mean, std — spot impossible values here

For missing values specifically:

df.isna().sum()           # count NaN per column
df.isnull().sum()         # same as isna() — both work
df[df["Phone"].notna()]   # rows that actually have a phone number

For duplicates and inconsistent categories:

df.duplicated()                         # True/False per row
df[df.duplicated()]                     # show only the duplicate rows
df["Paying_Customer"].value_counts()    # reveals: Y, N, Yes, y — all meaning the same thing

Always run describe() on numeric columns — the min and max alone will tell you immediately if something is wrong. A minimum age of -5 or a score of 110 out of 100 is a red flag you can't miss.

Step 2 — Fix Column Names First

Messy column names make everything harder. Clean them before anything else, so every subsequent operation uses predictable, consistent names.

Rename specific columns:

df.rename(columns={"Old Name": "New_Name", "phone number": "Phone"})

Clean all column names at once — the standard one-liner:

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

This strips surrounding spaces, converts to lowercase, and replaces spaces with underscores in a single chain. After this, "First Name" becomes "first_name", " Age " becomes "age", and so on.

Before:              After:
"First Name"    →    "first_name"
"  Age "        →    "age"
"Phone Number"  →    "phone_number"
"ZIP Code"      →    "zip_code"

Step 3 — Handle Missing Values

Missing values appear as NaN in Pandas. Before deciding what to do with them, you need to understand why they're missing — because the right fix depends on the cause.

Fill missing values with a fixed value:

df["Phone"].fillna("000-000-0000")    # placeholder for missing phone
df["Age"].fillna(0)                   # fill numeric with 0
df["Name"].fillna("Unknown")          # fill text with label
df["Score"].fillna(df["Score"].mean()) # fill with column average

Fill missing numeric values by estimating from neighbors:

df["Sales"] = df["Sales"].interpolate()

interpolate() fills NaN by calculating the midpoint between the values above and below it. This is ideal for time series or sensor data where values change gradually — it would be wrong to use for random datasets.

Before:     After interpolate():
100         100
NaN    →    150   ← estimated midpoint
200         200
NaN    →    225   ← estimated midpoint
250         250

Remove rows with missing values:

df.dropna()                            # drop any row with at least one NaN
df.dropna(subset=["Phone"])            # drop only if Phone is NaN
df.dropna(thresh=3)                    # drop rows that have fewer than 3 non-NaN values

inplace rule: use either inplace=True or variable assignment — never both. They do the same thing, and combining them produces a bug where df becomes None.
df.dropna(subset=["Phone"], inplace=True)   # ✅ modifies df directly
df = df.dropna(subset=["Phone"])            # ✅ returns new df
df = df.dropna(subset=["Phone"], inplace=True)  # ❌ df becomes None

Step 4 — Remove Duplicates

Duplicate rows silently inflate counts, averages, and totals. Always check for them.

df.drop_duplicates()                         # remove fully identical rows
df.drop_duplicates(subset=["Name"])          # remove rows with duplicate Name only
df.drop_duplicates(subset=["Name", "Email"]) # remove rows where both columns match
df.drop_duplicates(keep="last")              # keep the last occurrence instead of the first

Before:                        After drop_duplicates():
Name   Email                   Name   Email
Alice  alice@mail.com          Alice  alice@mail.com
Bob    bob@mail.com      →     Bob    bob@mail.com
Alice  alice@mail.com ← dup    Carol  carol@mail.com
Carol  carol@mail.com

Step 5 — Fix Data Types

Pandas sometimes reads numeric columns as strings, or dates as plain text. Operations on wrong types either crash or silently produce wrong results.

Change column type directly:

df["Zip_Code"] = df["Zip_Code"].astype(str)     # treat as text, not a number
df["Age"] = df["Age"].astype(int)               # convert to integer
df["Price"] = df["Price"].astype(float)         # convert to decimal

Handle messy numbers that contain symbols or text:

df["Price"] = pd.to_numeric(df["Price"], errors="coerce")

errors="coerce" converts anything that can't be parsed into NaN instead of crashing. This is the safe way to clean columns like "$1,200" or "N/A" that can't be directly cast with astype().

Before:           After pd.to_numeric(..., errors="coerce"):
"1200"    →       1200.0
"$850"    →       NaN     ← couldn't parse
"700.5"   →       700.5
"N/A"     →       NaN     ← couldn't parse

Convert to datetime:

df["Date"] = pd.to_datetime(df["Date"])
df["Date"] = pd.to_datetime(df["Date"], format="%d/%m/%Y")  # specify format if needed

Once a column is proper datetime, you unlock sorting by date, filtering by year/month, calculating time differences, and much more.

Step 6 — Clean Strings

String columns are almost always the messiest part of a real dataset. Pandas provides a full .str accessor with everything you need.

Strip unwanted characters from edges:

df["Name"] = df["Name"].str.strip()           # remove spaces from both sides
df["Name"] = df["Name"].str.strip(" ./-")     # remove spaces, dots, slashes, dashes
df["Name"] = df["Name"].str.lstrip()          # left side only
df["Name"] = df["Name"].str.rstrip()          # right side only

Replace characters anywhere in the string:

df["Phone"] = df["Phone"].str.replace(r"[a-zA-Z]", "", regex=True)   # remove all letters
df["Phone"] = df["Phone"].str.replace(r"\D", "", regex=True)          # keep digits only
df["Name"]  = df["Name"].str.replace(r"[./]", "", regex=True)         # remove dots and slashes

Standardize text case:

df["Name"] = df["Name"].str.lower()     # all lowercase → "alice"
df["Name"] = df["Name"].str.upper()     # all uppercase → "ALICE"
df["Name"] = df["Name"].str.title()     # title case    → "Alice"

Check text content:

df[df["Email"].str.contains("@")]           # filter valid-looking emails
df[df["Phone"].str.len() != 12]             # find rows where phone length is wrong
df[df["Name"].str.startswith("A")]          # names starting with A
df[df["Name"].str.endswith("e")]            # names ending with e

Extract a pattern using regex:

df["Zip"] = df["Address"].str.extract(r"(\d{5})")        # pull 5-digit zip code
df["Area_Code"] = df["Phone"].str.extract(r"^(\d{3})")   # pull first 3 digits

Split one column into multiple columns:

df[["Street", "State", "Zip"]] = (
    df["Address"]
    .str.split(",", n=2, expand=True)
)

n=2 limits the split to 2 times, producing exactly 3 parts regardless of how many commas the string contains. expand=True spreads those parts into separate columns.

Before:                              After:
"123 Main St, Texas, 75001"   →   Street="123 Main St"  State="Texas"  Zip="75001"

Step 7 — Replace and Standardize Values

Raw data often has the same value written multiple ways: "Y", "Yes", "yes", "y" — all meaning the same thing. Standardize them.

Replace whole cell values using a dictionary:

df["Status"] = df["Status"].replace({"Y": "Yes", "N": "No", "y": "Yes"})

# Apply to multiple columns at once
df[["Paying_Customer", "Do_Not_Contact"]] = (
    df[["Paying_Customer", "Do_Not_Contact"]]
    .replace({"Y": "Yes", "N": "No", "y": "Yes", "n": "No"})
)

Overwrite invalid values using loc[]:

df.loc[df["Age"] < 0,   "Age"] = None     # negative age → invalid
df.loc[df["Age"] > 120, "Age"] = None     # age over 120 → invalid
df.loc[df["Price"] < 0, "Price"] = None   # negative price → invalid

Keep valid values, replace everything else with NaN:

df["Salary"] = df["Salary"].where(df["Salary"] > 0)
# Rows where Salary <= 0 become NaN; the rest are untouched

Step 8 — Handle Outliers

An outlier is a value that exists but shouldn't — a score of 110/100, an age of 200, a price of -50. clip() is the clean way to enforce valid ranges.

Cap values within a valid range:

df["Score"] = df["Score"].clip(0, 100)      # 110 → 100, -10 → 0
df["Age"]   = df["Age"].clip(0, 120)        # 200 → 120, -5  → 0
df["Age"]   = np.clip(df["Age"], 0, 120)    # NumPy version — same result

Before:   After clip(0, 100):
85        85
110   →   100   ← capped at max
-10   →   0     ← capped at min
72        72

clip() is gentler than dropping rows — it preserves the row but brings the outlier within the valid boundary. Use it when the row has useful data in other columns that you don't want to lose.

Step 9 — Drop Rows and Columns

Sometimes the cleanest fix is simply to remove what you don't need.

Drop rows by index:

df.drop(0)              # remove row at index 0
df.drop([0, 2, 5])      # remove multiple rows by index

Drop columns:

df.drop(columns=["Age", "Phone"])      # preferred style
df.drop("Age", axis=1)                 # axis=1 means "columns"
df.drop(["Age", "Phone"], axis=1)      # multiple columns

Step 10 — Create New Columns

Cleaning isn't only about removing bad data — it's also about building the right structure for analysis.

Fixed value:

df["Source"] = "CSV Import"    # tag every row with its data source

Calculated from another column:

df["Price_With_Tax"] = df["Price"] * 1.15
df["Name_Length"]    = df["Name"].str.len()

Based on a condition:

df["Status"] = np.where(df["Age"] >= 18, "Adult", "Minor")

Based on multiple conditions:

conditions = [df["Score"] >= 90, df["Score"] >= 70, df["Score"] >= 50]
choices    = ["A", "B", "C"]
df["Grade"] = np.select(conditions, choices, default="F")

Custom logic with apply():

df["Name"] = df["Name"].apply(lambda x: x.title() if isinstance(x, str) else x)

def categorize_price(price):
    if price > 1000: return "Expensive"
    elif price > 500: return "Medium"
    else:            return "Cheap"

df["Category"] = df["Price"].apply(categorize_price)

Step 11 — Add New Rows

Add a single row using concat():

new_row = pd.DataFrame([{"Name": "Frank", "Age": 28, "Phone": "055-111-2222"}])
df = pd.concat([df, new_row], ignore_index=True)

Add multiple rows at once:

new_rows = pd.DataFrame([
    {"Name": "Grace", "Age": 31},
    {"Name": "Hank",  "Age": 24},
])
df = pd.concat([df, new_rows], ignore_index=True)

ignore_index=True resets the index after concatenation so there are no gaps or repeated numbers:

Without ignore_index:    With ignore_index=True:
0  Alice                 0  Alice
1  Bob          →        1  Bob
0  Grace                 2  Grace   ← clean
1  Hank                  3  Hank    ← clean

Step 12 — Reset the Index

After dropping rows, the index has gaps. Reset it before analysis or before passing data to another function.

df = df.reset_index(drop=True)

drop=True discards the old index numbers entirely. drop=False keeps the old index as a new column — useful if you need to trace where a row originally came from.

Putting It All Together — A Real Cleaning Pipeline

Here's what a complete cleaning workflow looks like from start to finish:

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")

# 1. Inspect
print(df.info())
print(df.isna().sum())
print(df.describe())

# 2. Fix column names
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

# 3. Drop useless columns and duplicates
df.drop(columns=["not_useful"], inplace=True)
df.drop_duplicates(inplace=True)

# 4. Handle missing values
df.dropna(subset=["phone_number"], inplace=True)
df["email"].fillna("unknown@unknown.com", inplace=True)
df["sales"] = df["sales"].interpolate()

# 5. Fix data types
df["date"]     = pd.to_datetime(df["date"])
df["price"]    = pd.to_numeric(df["price"], errors="coerce")
df["zip_code"] = df["zip_code"].astype(str)

# 6. Clean strings
df["name"] = df["name"].str.strip().str.title()
df["phone_number"] = (
    df["phone_number"]
    .str.replace(r"\D", "", regex=True)
    .str.replace(r"(\d{3})(\d{3})(\d{4})", r"\1-\2-\3", regex=True)
    .replace("", "000-000-0000")
)

# 7. Fix invalid values and outliers
df.loc[df["age"] < 0, "age"] = None
df["score"] = df["score"].clip(0, 100)

# 8. Standardize categories
df["status"] = df["status"].replace({"Y": "Yes", "N": "No", "y": "Yes"})

# 9. Filter rows
df = df[df["do_not_contact"] != "Yes"]

# 10. Reset index
df = df.reset_index(drop=True)

print(df.info())
print(df.head())

Now Combine — Merging & Joining DataFrames

Once each source is clean, the next challenge is combining them. In the real world, data almost never lives in a single file. You might have orders in one table, customers in another, and products in a third. Merging is how you bring those pieces together.

Pandas provides three main tools for this:

Tool	What it does
`merge()`	Joins two DataFrames by matching on a shared column
`concat()`	Stacks DataFrames vertically or side by side
`join()`	Joins on the index instead of a column

We'll use these two DataFrames for all the examples:

customers = pd.DataFrame({
    "ID":        [1, 2, 3, 4],
    "Name":      ["Alice", "Bob", "Carol", "Dave"],
    "Country":   ["USA", "UK", "India", "USA"],
})

orders = pd.DataFrame({
    "ID":        [1, 2, 3, 5],
    "Product":   ["Laptop", "Phone", "Tablet", "Monitor"],
    "Price":     [1200, 800, 600, 400],
})

Notice that ID=4 (Dave) exists in customers but not in orders, and ID=5 exists in orders but not in customers. This is exactly the kind of mismatch you encounter in practice — and why the how= parameter matters so much.

1️⃣ `merge()` — The Main Tool

merge() works like a SQL JOIN. It aligns two DataFrames by finding rows where a shared column has the same value.

Basic syntax:

customers.merge(orders, how="inner", on="ID")

The `how=` Parameter — All Four Types

# INNER → only rows that match in both DataFrames (default)
customers.merge(orders, how="inner", on="ID")
# Result: IDs 1, 2, 3 only — Dave (4) and Monitor (5) are excluded

# LEFT → all rows from the left + matched rows from the right
customers.merge(orders, how="left", on="ID")
# Result: all 4 customers — Dave gets NaN for Product and Price

# RIGHT → all rows from the right + matched rows from the left
customers.merge(orders, how="right", on="ID")
# Result: all 4 orders — Monitor (ID=5) gets NaN for Name and Country

# OUTER → all rows from both, NaN where no match
customers.merge(orders, how="outer", on="ID")
# Result: all 5 unique IDs — gaps filled with NaN on both sides

`how=`	Keeps from Left	Keeps from Right	NaN Possible
`inner`	Matched only	Matched only	No
`left`	All rows	Matched only	Right side
`right`	Matched only	All rows	Left side
`outer`	All rows	All rows	Both sides

CROSS merge — every row with every row:

customers.merge(orders, how="cross")
# Produces 4 × 4 = 16 rows — every customer paired with every order
# No on= parameter — it will raise an error if you include it

The `on=` Parameter — What to Match On

# Match on a single column
customers.merge(orders, how="inner", on="ID")

# Match on multiple columns — both must match simultaneously
customers.merge(orders, how="inner", on=["ID", "Country"])

# Auto-detect — Pandas finds all shared column names automatically
customers.merge(orders)

Be careful with auto-detect. If both DataFrames happen to share a column that you don't intend to join on (like a generic "Name" column), Pandas will include it in the match condition and produce unexpected results. Explicit is safer.

Handling Shared Column Names — `_x` and `_y`

If both DataFrames have a column with the same name, but you only join on ID, Pandas keeps both copies and renames them automatically:

# Both DataFrames have a "Name" column, but you only merge on "ID"
customers.merge(orders, how="inner", on="ID")
# Result: Name_x (from customers), Name_y (from orders)

Three ways to fix this:

# Option 1 — merge on all shared columns so no duplicates appear
customers.merge(orders, how="inner", on=["ID", "Name"])

# Option 2 — use custom suffixes instead of _x and _y
customers.merge(orders, how="inner", on="ID", suffixes=("_customer", "_order"))

# Option 3 — drop the shared column from one DataFrame before merging
orders_clean = orders.drop(columns=["Name"])
customers.merge(orders_clean, how="inner", on="ID")

Other Useful Parameters

# left_on / right_on — when the key column has different names in each DataFrame
customers.merge(orders, how="inner", left_on="CustomerID", right_on="OrderID")

# indicator — adds a column showing where each row came from
customers.merge(orders, how="outer", on="ID", indicator=True)
# _merge column values: "left_only", "right_only", "both"

The indicator=True parameter is especially useful for auditing: after a merge, you can immediately see which rows matched, which came only from the left, and which came only from the right.

2️⃣ `concat()` — Stacking DataFrames

While merge() combines DataFrames horizontally by matching values, concat() simply stacks them — either rows on top of rows, or columns side by side.

pd.concat([df1, df2])        # stack vertically (default)
pd.concat([df1, df2], axis=1)  # stack horizontally

`axis=` Parameter

# axis=0 → stack rows vertically (append one DataFrame below another)
monthly_data = pd.concat([january, february, march], axis=0)

# axis=1 → stack columns side by side
combined = pd.concat([names_df, scores_df], axis=1)

`join=` Parameter

# outer (default) → keep all columns, NaN where a column doesn't exist in one source
pd.concat([df1, df2], join="outer")

# inner → keep only columns that exist in all DataFrames
pd.concat([df1, df2], join="inner")

`ignore_index=` Parameter

# Without ignore_index — original indices are preserved, which can repeat
pd.concat([df1, df2])
# Index: 0, 1, 2, 0, 1, 2  ← 0, 1, 2 repeats

# With ignore_index=True — fresh sequential index
pd.concat([df1, df2], ignore_index=True)
# Index: 0, 1, 2, 3, 4, 5  ← clean

Always use ignore_index=True after stacking unless you specifically need the original indices.

`keys=` Parameter

# Label each source so you can identify where each row came from
combined = pd.concat([df1, df2], keys=["source_a", "source_b"])

This creates a MultiIndex where the first level is your label and the second is the original row index — useful when you need to trace data back to its source after combining.

`verify_integrity=` Parameter

# Raises an error if any duplicate index values are found after concatenation
pd.concat([df1, df2], verify_integrity=True)

`merge()` vs `concat()` — When to Use Which

	`merge()`	`concat()`
Direction	Horizontal (by matching)	Vertical or horizontal (by stacking)
Needs a shared key column	Yes	No
Best for	Joining related tables	Combining same-structure data
Equivalent to	SQL JOIN	SQL UNION

A simple rule: if you're combining columns from different tables, use merge(). If you're stacking more rows of the same structure, use concat().

3️⃣ `join()` — Index-Based Joining

join() works like merge(), but it joins on the index instead of a regular column. You need to set the index first.

customers_indexed = customers.set_index("ID")
orders_indexed    = orders.set_index("ID")

customers_indexed.join(orders_indexed, how="inner")

The how= options are the same as merge(): inner, left, right, outer.

Why `merge()` Is Almost Always Better

	`merge()`	`join()`
Join on any column	✅ Yes	❌ No — index only
Join on index	✅ Yes	✅ Yes
Requires `set_index()` first	❌ No	✅ Yes
Simpler syntax	✅ Yes	⚠️ Extra steps

Use join() only when your data is already indexed — for example, after a groupby() that produced a meaningful index. Otherwise, merge() is cleaner and more flexible.

Common Patterns

Clean first, then merge

The most important workflow: clean each source independently before combining them.

# Clean customers
customers["Name"] = customers["Name"].str.strip().str.title()
customers = customers.drop_duplicates(subset=["ID"])

# Clean orders
orders["Price"] = pd.to_numeric(orders["Price"], errors="coerce")
orders = orders.dropna(subset=["Price"])

# Now merge
result = customers.merge(orders, how="left", on="ID")

Merge and immediately filter

# Keep only rows that matched in both DataFrames (inner), then filter by price
result = customers.merge(orders, how="inner", on="ID")
result = result[result["Price"] > 500]

Combine monthly files with concat, then analyze

import glob

all_files = glob.glob("sales_*.csv")
monthly_dfs = [pd.read_csv(f) for f in all_files]
full_year = pd.concat(monthly_dfs, ignore_index=True)

# Now clean and analyze the full year
full_year["Date"] = pd.to_datetime(full_year["Date"])
full_year.drop_duplicates(inplace=True)

Complete Summary Table

Function	Category	Purpose
`df.info()`	Inspection	Column types and null counts
`df.describe()`	Inspection	Statistical summary
`df.isna().sum()`	Inspection	Count missing values per column
`df.value_counts()`	Inspection	Frequency of each unique value
`df.duplicated()`	Inspection	Identify duplicate rows
`df.fillna()`	Missing Values	Fill NaN with a value
`df.dropna()`	Missing Values	Remove rows with NaN
`df.interpolate()`	Missing Values	Estimate NaN from neighbors
`df.drop_duplicates()`	Duplicates	Remove duplicate rows
`df.drop()`	Structure	Remove rows or columns
`df.rename()`	Structure	Rename specific columns
`df.columns.str.*`	Structure	Clean all column names at once
`df.astype()`	Types	Change column data type
`pd.to_numeric()`	Types	Convert to number, coerce errors
`pd.to_datetime()`	Types	Convert to datetime
`str.strip()`	Strings	Remove characters from edges
`str.replace()`	Strings	Replace characters in text
`str.split()`	Strings	Split into multiple columns
`str.extract()`	Strings	Extract regex pattern
`str.contains()`	Strings	Filter rows by text match
`str.len()`	Strings	Get text length per row
`df.replace()`	Values	Replace whole cell values
`df.loc[cond, col]`	Values	Overwrite values matching condition
`df.where()`	Values	Keep valid values, NaN the rest
`df.clip()`	Outliers	Cap values within a valid range
`np.where()`	New Columns	Column based on one condition
`np.select()`	New Columns	Column based on multiple conditions
`df.apply()`	New Columns	Custom function per row or column
`df.reset_index()`	Structure	Reset index after dropping rows
`merge()`	Combining	Join two DataFrames by matching column
`concat()`	Combining	Stack DataFrames vertically or horizontally
`join()`	Combining	Join two DataFrames by index

This is Part 3 of the Pandas for Data Science series. Next up: Part 4 — Data Visualization with Matplotlib & Seaborn

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw

Mastering Pandas — Part 2: GroupBy & Indexing

Hussein Mahdi — Wed, 11 Mar 2026 07:04:00 +0000

Pandas for Data Science Series — Article #2

From Loading to Understanding

In Part 1, you learned how to load data from virtually any source and get your first look at its structure. Now it's time to go deeper — to ask real questions about your data and retrieve exactly what you need from it.

This article covers two fundamental skill sets that every data analyst uses daily:

GroupBy — splitting your data into groups and summarizing each one independently
Indexing — selecting exactly the rows and columns you need, with precision

These two topics are deeply connected in practice. You group data to understand it at a high level, and you index into it to examine the details. Together, they form the core of exploratory data analysis.

We'll use this sample DataFrame throughout all the examples:

import pandas as pd

data = {
    'country':   ['USA', 'USA', 'UK', 'UK', 'India', 'India'],
    'gender':    ['Male', 'Female', 'Male', 'Female', 'Male', 'Female'],
    'product':   ['Laptop', 'Phone', 'Laptop', 'Tablet', 'Phone', 'Laptop'],
    'price':     [1200, 800, 1100, 600, 700, 950],
    'quantity':  [3, 5, 2, 4, 6, 3],
    'rating':    [4.5, 4.2, 4.8, 3.9, 4.1, 4.6]
}
df = pd.DataFrame(data)

Part 1 — GroupBy

groupby() is one of the most powerful operations in Pandas. It follows a simple but flexible pattern: split the data into groups based on a column's values, apply a function to each group independently, and combine the results back into a single output. This is known as the Split → Apply → Combine pattern.

# The fundamental pattern
df.groupby("column")["target"].function()

1️⃣ Aggregation Functions — The Core of GroupBy

Aggregation functions collapse each group into a single summary value. These are the most commonly used GroupBy operations in day-to-day data work.

df.groupby("gender")["price"].mean()     # average price per gender
df.groupby("gender")["price"].sum()      # total revenue per gender
df.groupby("gender")["price"].count()    # number of transactions per gender
df.groupby("gender")["price"].min()      # lowest price per gender
df.groupby("gender")["price"].max()      # highest price per gender
df.groupby("gender")["price"].median()   # middle value per gender
df.groupby("gender")["price"].std()      # standard deviation per gender
df.groupby("gender")["price"].var()      # variance per gender

2️⃣ `agg()` — Multiple Functions at Once

Instead of calling one function at a time, agg() lets you apply several aggregations in a single step. You can use the default names or assign your own custom labels — the second form is preferred in professional work because the output is immediately readable.

# Default names
df.groupby("gender")["price"].agg(["mean", "sum", "min", "max"])

# Custom names — preferred for readability
df.groupby("gender")["price"].agg(
    Average = "mean",
    Total   = "sum",
    Lowest  = "min",
    Highest = "max"
)

3️⃣ `transform()` — Keep the Original Shape

Unlike aggregation functions that collapse groups into fewer rows, transform() returns a result with the same shape as the original DataFrame. This makes it ideal for adding a new column that reflects group-level statistics back onto each individual row.

# Add each row's group average back to the original DataFrame
df["gender_avg_price"] = df.groupby("gender")["price"].transform("mean")

This is especially useful for normalization, group-relative comparisons, and feature engineering in machine learning pipelines.

4️⃣ `apply()` — Custom Logic Per Group

When the built-in functions aren't enough, apply() lets you pass any custom function — including lambdas — and run it independently on each group. This gives you full flexibility.

# Range (max - min) of price per gender
df.groupby("gender")["price"].apply(lambda x: x.max() - x.min())

Note: apply() is the most flexible option but also the slowest. For standard aggregations, always prefer the built-in functions or agg() — they are significantly faster on large datasets.

5️⃣ `filter()` — Keep or Drop Entire Groups

filter() evaluates a condition for each group and keeps only the groups where the condition is True. Unlike boolean indexing which filters individual rows, filter() operates at the group level — it keeps or drops entire groups together.

# Keep only gender groups where the average price exceeds 200
df.groupby("gender").filter(lambda x: x["price"].mean() > 200)

6️⃣ `size()` vs `count()` — A Critical Distinction

These two functions look similar but behave differently, and confusing them can lead to subtle bugs.

df.groupby("gender").size()    # counts ALL rows per group, including NaN values
df.groupby("gender").count()   # counts only NON-NULL values per column per group

	`size()`	`count()`
Counts NaN	✅ Yes	❌ No
Returns	Single Series	DataFrame (one column per column)
Use when	You want total rows	You want non-missing values

Use size() when you want to know how many rows exist in each group. Use count() when you want to know how much valid (non-missing) data each group has.

7️⃣ `nunique()` — Count Distinct Values Per Group

Returns the number of unique values in a column within each group. Useful for understanding variety and diversity within categories.

# How many unique products does each country sell?
df.groupby("country")["product"].nunique()

8️⃣ `first()` & `last()` — Boundary Rows of Each Group

Returns the first or last non-null row from each group. The result preserves the original row order within each group.

df.groupby("gender").first()    # first row of each gender group
df.groupby("gender").last()     # last row of each gender group

9️⃣ Group by Multiple Columns

You can group by more than one column at a time by passing a list. The result has a MultiIndex — one level per grouping column.

# Average price broken down by both country and gender
df.groupby(["country", "gender"])["price"].mean()

GroupBy Summary Table

Function	Purpose
`mean()`	Average value per group
`sum()`	Total value per group
`count()`	Count of non-null values per group
`size()`	Count of all rows per group (including NaN)
`min()` / `max()`	Minimum and maximum per group
`median()`	Middle value per group
`std()` / `var()`	Spread of data per group
`agg()`	Multiple aggregations in one step
`apply()`	Custom function per group
`transform()`	Group result mapped back to original shape
`filter()`	Keep or drop entire groups by condition
`nunique()`	Count of distinct values per group
`first()` / `last()`	First and last row of each group

Part 2 — Indexing

Indexing is how you reach into your DataFrame and retrieve exactly what you need — a single value, a slice of rows, a set of columns, or a filtered subset. Pandas offers several indexing tools, each suited to a different situation.

Understanding which tool to use — and why — is one of the most important skills you can build as a data analyst.

1️⃣ `[]` — Basic Indexing

The simplest form of indexing. Use it for quick column selection or row slicing by position.

df["price"]                    # select a single column → returns a Series
df[["price", "quantity"]]      # select multiple columns → returns a DataFrame
df[0:3]                        # select rows by slice (first 3 rows)

Note: Using [] for row selection only works with slices (df[0:3]), not single integers (df[0] will raise an error). For precise row access, use loc[] or iloc[].

2️⃣ `loc[]` — Label-Based Indexing

loc[] selects data by label — that is, by the actual index values and column names. It is the preferred tool when you know the names of what you're looking for.

df.loc[0]                         # row with index label 0
df.loc[0:3]                       # rows with index labels 0 through 3 (inclusive)
df.loc[0, "price"]                # single value: row 0, column "price"
df.loc[0:3, "price":"rating"]     # rows 0–3, columns from "price" to "rating"
df.loc[df["gender"] == "Male"]    # all rows where gender is Male

Important: When using loc[] with a slice (0:3), both endpoints are inclusive. This differs from standard Python slicing where the end is exclusive.

3️⃣ `iloc[]` — Position-Based Indexing

iloc[] selects data by integer position — the physical row and column numbers, regardless of what the index or column names are. It behaves like standard Python list indexing.

df.iloc[0]           # first row (position 0)
df.iloc[-1]          # last row
df.iloc[0:3]         # first 3 rows (positions 0, 1, 2 — end is exclusive)
df.iloc[0, 2]        # row at position 0, column at position 2
df.iloc[0:3, 0:3]    # first 3 rows, first 3 columns
df.iloc[:, 0]        # all rows, first column only

`loc[]` vs `iloc[]` — The Key Distinction

This is the most important comparison in Pandas indexing. Choosing the wrong one is a common source of bugs.

	`loc[]`	`iloc[]`
Based on	Labels (names)	Integer positions
End of slice	Inclusive	Exclusive
Use when	You know column/index names	You know position numbers
Works after reset_index	✅ Yes	✅ Yes
Fragile after sort/filter	⚠️ If index changes	✅ Always positional

# These may return DIFFERENT rows if the index is not 0,1,2,3...
df.loc[2]     # row where index label == 2
df.iloc[2]    # third row regardless of its index label

4️⃣ `at[]` & `iat[]` — Fast Single Value Access

When you need to read or write a single cell, at[] and iat[] are faster than loc[] and iloc[] because they are optimized specifically for scalar access.

# at[] — by label
df.at[0, "price"]              # get value at row 0, column "price"
df.at[0, "price"] = 999        # set value at row 0, column "price"

# iat[] — by position
df.iat[0, 3]                   # get value at row 0, column position 3
df.iat[0, 3] = 999             # set value at row 0, column position 3

Use at[] / iat[] inside loops or any time you are accessing one cell at a time — the performance difference becomes significant at scale.

5️⃣ Boolean Indexing — Filter by Condition

Boolean indexing is the standard way to filter rows based on a condition. It works by creating a True/False mask that Pandas applies to the DataFrame.

df[df["price"] > 900]                                         # price above 900
df[df["gender"] == "Female"]                                  # only Female rows
df[df["country"] == "USA"]                                    # only USA rows

# Combining conditions
df[(df["price"] > 900) & (df["gender"] == "Female")]          # AND
df[(df["price"] > 900) | (df["country"] == "UK")]             # OR
df[~(df["gender"] == "Male")]                                  # NOT (negation)

Always wrap each condition in parentheses when combining with &, |, or ~. Python's operator precedence will cause unexpected results without them.

6️⃣ `isin()` — Match Against a List of Values

isin() checks whether each value in a column belongs to a provided list. It is the clean, readable alternative to chaining multiple OR conditions.

df[df["country"].isin(["USA", "UK", "India"])]    # keep these countries
df[~df["country"].isin(["USA", "UK"])]            # exclude these countries

7️⃣ `query()` — Filter Using a String Expression

query() lets you filter rows using a readable string expression instead of boolean masks. It is especially useful for complex conditions — the syntax is cleaner and closer to natural language.

df.query("price > 900")
df.query("gender == 'Female'")
df.query("price > 900 and gender == 'Female'")
df.query("country in ['USA', 'UK']")

query() produces the same result as boolean indexing — it is purely a readability preference. For simple one-line filters, boolean indexing is fine. For multi-condition filters, query() is often easier to read and maintain.

8️⃣ `where()` — Conditional Value Replacement

Unlike the filtering tools above that remove rows, where() keeps all rows but replaces values that don't meet the condition with NaN or a value you specify.

df["price"].where(df["price"] > 900)           # keep prices > 900, rest → NaN
df["price"].where(df["price"] > 900, other=0)  # keep prices > 900, rest → 0

This is useful when you want to preserve the DataFrame's shape while masking out values that don't meet a criterion.

9️⃣ `set_index()` — Use a Column as the Index

By default, Pandas assigns a numeric index (0, 1, 2, ...). set_index() lets you replace that with a meaningful column — such as a country name or an order ID — making label-based indexing more intuitive.

df.set_index("country")                        # use country as the index

Understanding inplace: This parameter controls whether the operation modifies the original DataFrame or returns a new one.

# inplace=False (default) — original is unchanged, result is a new DataFrame
df2 = df.set_index("country")

# inplace=True — original DataFrame is modified directly
df.set_index("country", inplace=True)

Most Pandas functions that modify structure accept inplace. The default is always False, meaning a new DataFrame is returned unless you explicitly ask for in-place modification.

🔟 `xs()` — Cross-Section for MultiIndex

After grouping by multiple columns, the resulting DataFrame has a MultiIndex — a hierarchical index with one level per grouping column. xs() (cross-section) lets you select a slice at a specific level of that index cleanly.

# After grouping by country and gender
grouped = df.groupby(["country", "gender"])["price"].mean().reset_index()
grouped = grouped.set_index(["country", "gender"])

# Select all rows where gender == "Female" across all countries
grouped.xs("Female", level="gender")

Indexing Summary Table

Tool	Based On	Best For
`[]`	Label	Quick column select or row slice
`loc[]`	Label	Rows & columns by name
`iloc[]`	Position	Rows & columns by number
`at[]`	Label	Single cell — fast read/write
`iat[]`	Position	Single cell — fast read/write
Boolean `[]`	Condition	Filter rows by expression
`isin()`	List	Match multiple values at once
`query()`	String	Readable multi-condition filtering
`where()`	Condition	Replace non-matching values
`set_index()`	Column	Use meaningful column as index
`xs()`	Label	MultiIndex cross-section

Most Important Rules to Remember

loc[]    →  when you know column NAMES  (slice end is inclusive)
iloc[]   →  when you know column NUMBERS (slice end is exclusive)
query()  →  when you want clean, readable filter expressions
Boolean  →  when you want to filter rows by condition
at/iat[] →  when you need a single cell value — fastest option

Common Patterns

Group, then Index into the Result

The most common real-world flow: summarize by group, then retrieve a specific slice from the result.

# Average price per country, then select only the top 2
df.groupby("country")["price"].mean().nlargest(2)

Add a Group-Level Column, then Filter

Use transform() to add group statistics back to the original DataFrame, then filter using that new column.

# Flag rows where price exceeds the group average for their gender
df["gender_avg"] = df.groupby("gender")["price"].transform("mean")
df[df["price"] > df["gender_avg"]]

Combine GroupBy with `agg()` and `query()`

# Total and average price per country, then keep only countries with avg > 800
result = df.groupby("country")["price"].agg(Total="sum", Average="mean")
result.query("Average > 800")

Complete Summary Table

Function	Category	Purpose
`mean()` / `sum()` / `count()`	GroupBy	Core aggregations per group
`min()` / `max()` / `median()`	GroupBy	Range and center per group
`std()` / `var()`	GroupBy	Spread of data per group
`agg()`	GroupBy	Multiple aggregations in one call
`apply()`	GroupBy	Custom function per group
`transform()`	GroupBy	Group result mapped to original shape
`filter()`	GroupBy	Keep or drop entire groups
`size()` / `nunique()`	GroupBy	Row count / unique value count per group
`first()` / `last()`	GroupBy	Boundary rows of each group
`[]`	Indexing	Quick column or slice access
`loc[]`	Indexing	Label-based row & column selection
`iloc[]`	Indexing	Position-based row & column selection
`at[]` / `iat[]`	Indexing	Fast single cell access
Boolean indexing	Indexing	Filter rows by condition
`isin()`	Indexing	Match a list of values
`query()`	Indexing	Readable string-based filtering
`where()`	Indexing	Replace non-matching values
`set_index()`	Indexing	Promote a column to the index
`xs()`	Indexing	MultiIndex cross-section

This is Part 2 of the Pandas for Data Science series. Next up: Part 3 — Data Cleaning & Merging/Joining .

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw

Mastering Pandas — Part 1: Reading, Sorting & Displaying Data

Hussein Mahdi — Thu, 26 Feb 2026 08:38:00 +0000

Pandas for Data Science Series — Article #1

What is Pandas and Why Does It Matter?

Pandas is an open-source Python library built specifically for data manipulation and analysis. Released in 2008 and named after "Panel Data" — a term from econometrics — it has since become one of the most essential tools in the entire Python ecosystem.

At its core, Pandas gives you two powerful data structures:

Series — a single column of data
DataFrame — a full table with rows and columns (think of it as a turbocharged spreadsheet you control with code)

Whether you're working in data science, machine learning, financial analysis, or business intelligence, chances are you'll be loading, exploring, and transforming data with Pandas before you do anything else. It integrates seamlessly with libraries like NumPy, Matplotlib, Seaborn, Scikit-learn, and TensorFlow, making it the essential starting point in nearly every data workflow.

In this first article of the series, you'll cover two essential skill sets: reading data into Pandas from virtually any file format, and sorting and displaying that data in ways that let you understand it quickly.

Let's get started 🎯

Part 1 — Reading Files into Pandas

Before you can analyze anything, you need to load your data. Pandas makes this simple with a family of read_*() functions — one for almost every file format you'll encounter in the real world.

CSV File — Most Common

CSV (Comma-Separated Values) files are the bread and butter of data work. read_csv() is almost certainly the first Pandas function you'll use in any project.

import pandas as pd

df = pd.read_csv("file.csv")                          # basic read
df = pd.read_csv("file.csv", header=0)                # first row as header
df = pd.read_csv("file.csv", index_col="order_id")    # set column as index
df = pd.read_csv("file.csv", nrows=100)               # read only first 100 rows
df = pd.read_csv("file.csv", skiprows=2)              # skip first 2 rows
df = pd.read_csv("file.csv", usecols=["price","quantity"])  # specific columns only
df = pd.read_csv("file.csv", na_values=["NA","?"])    # define missing values
df = pd.read_csv("file.csv", sep=";")                 # semicolons instead of commas

Excel File

Excel files are ubiquitous in business environments. read_excel() handles them cleanly, and you can target specific sheets by name or position.

df = pd.read_excel("file.xlsx")                       # basic read
df = pd.read_excel("file.xlsx", sheet_name="Sheet1")  # specific sheet by name
df = pd.read_excel("file.xlsx", sheet_name=0)         # first sheet by index
df = pd.read_excel("file.xlsx", skiprows=2)           # skip first 2 rows
df = pd.read_excel("file.xlsx", usecols="A:D")        # read columns A to D

JSON File

JSON is the standard format for web APIs. read_json() can parse various JSON structures, including a flat list of records.

df = pd.read_json("file.json")                        # basic read
df = pd.read_json("file.json", orient="records")      # list of records format

HTML File — Web Scraping Made Easy

One of Pandas' more surprising capabilities: it can extract tables directly from HTML pages, including live websites. Simple web scraping with zero extra libraries.

tables = pd.read_html("file.html")                          # returns list of all tables
df     = pd.read_html("https://website.com/table")[0]       # first table from a URL

SQL Database

When your data lives in a database, Pandas can query it directly. You bring a connection object and a SQL query, and it returns a DataFrame.

import sqlite3

conn = sqlite3.connect("database.db")

df = pd.read_sql("SELECT * FROM table", conn)    # full SQL query
df = pd.read_sql_table("table_name", conn)       # read entire table directly

Text File

read_table() is the text-file counterpart to read_csv(). The only real difference is its default separator — a tab instead of a comma — but it accepts all the same parameters.

df = pd.read_table("file.txt")                      # tab separated (default)
df = pd.read_table("file.txt", sep=",")             # comma separated
df = pd.read_table("file.txt", sep=";")             # semicolon separated
df = pd.read_table("file.txt", sep="|")             # pipe separated
df = pd.read_table("file.txt", header=None)         # file has no header row
df = pd.read_table("file.txt", names=["col1","col2"])  # add column names manually
df = pd.read_table("file.txt", skiprows=2)          # skip first 2 rows
df = pd.read_table("file.txt", nrows=100)           # read only 100 rows
df = pd.read_fwf("file.txt")                        # fixed-width text file

read_csv() vs read_table() — The only practical difference is the default separator: read_csv() uses a comma (,) while read_table() uses a tab (\t). Every other parameter works identically in both.

	`read_csv()`	`read_table()`
Default separator	`,` comma	`\t` tab
File type	CSV files	TXT files
Speed	Same	Same
Parameters	Same	Same

Other Formats

# Clipboard — copy any table, then run this
df = pd.read_clipboard()

# Parquet — the preferred format for big data (very fast)
df = pd.read_parquet("file.parquet")

# XML
df = pd.read_xml("file.xml")

Summary — All Read Functions at a Glance

Function	File Type	Common Use
`read_csv()`	CSV	Most common — daily use
`read_excel()`	XLSX	Excel files from business
`read_json()`	JSON	Web APIs and REST data
`read_html()`	HTML	Web scraping tables
`read_sql()`	Database	SQL queries
`read_table()`	TXT	Tab-separated text files
`read_clipboard()`	Clipboard	Quick copy-paste workflow
`read_parquet()`	Parquet	Large-scale / big data
`read_xml()`	XML	Structured XML data

Most Important Parameters — Work With ALL Functions

nrows      →  how many rows to read
skiprows   →  how many rows to skip at the top
usecols    →  which columns to read
index_col  →  which column to use as index
na_values  →  what to treat as missing value (NaN)
header     →  which row to use as column header
sep        →  what separator/delimiter to use

Part 2 — Sorting and Displaying Data

Once your data is loaded, the first thing you want to do is understand it. The functions in this section are your primary tools for exploring structure, distributions, relationships, and ordering — before you write a single line of analysis.

We'll use this sample DataFrame throughout all the examples:

import pandas as pd

data = {
    'Country':          ['China', 'India', 'USA', 'Brazil', 'UK'],
    'Continent':        ['Asia', 'Asia', 'North America', 'South America', 'Europe'],
    '2022 Population':  [1412000000, 1380000000, 331000000, 214000000, 67000000],
    'Area (km2)':       [9597000, 3287000, 9834000, 8516000, 243000]
}
df = pd.DataFrame(data)

Sorting Functions

1. sort_values()

The most commonly used sorting function. Reorders the DataFrame by the values in one or more columns. By default it sorts ascending; pass ascending=False to flip it.

# Ascending order (default)
df.sort_values('2022 Population')

# Descending order
df.sort_values('2022 Population', ascending=False)

# Sort by multiple columns: Continent A→Z, then Population largest first
df.sort_values(['Continent', '2022 Population'], ascending=[True, False])

2. sort_index()

Sorts by the DataFrame's index rather than column values. This becomes important after filtering, merging, or shuffling rows, which can leave the index disordered.

df.sort_index()                # sort index ascending
df.sort_index(ascending=False) # sort index descending

3. nlargest()

Returns the top N rows with the largest values in a specified column. More efficient and more readable than combining sort_values() with head().

# Top 3 most populated countries
df.nlargest(3, '2022 Population')

4. nsmallest()

Returns the top N rows with the smallest values in a specified column.

# 3 smallest countries by area
df.nsmallest(3, 'Area (km2)')

5. rank()

Assigns a rank to each row based on a column's values, without changing the row order. Useful for adding a ranking column to your existing DataFrame.

# Add a population rank column (1 = largest)
df['Population Rank'] = df['2022 Population'].rank(ascending=False)

Displaying Functions

1. head() & tail()

The simplest inspection tools. head() shows the first N rows (default 5) and tail() shows the last N rows. These are usually the first things you run on any new dataset.

df.head()    # first 5 rows
df.head(10)  # first 10 rows
df.tail()    # last 5 rows
df.tail(3)   # last 3 rows

2. sample()

Returns N randomly selected rows. Ideal for getting a representative look at a large dataset where the first or last rows might not be representative.

df.sample(3)  # 3 random rows

3. info()

Displays the DataFrame's structure: column names, data types, number of non-null values, and memory usage. This should be one of the very first functions you call on any new dataset — it immediately reveals missing data and wrong data types.

df.info()

4. describe()

Returns a statistical summary for all numeric columns: count, mean, standard deviation, min, max, and quartiles. A quick way to understand the scale and distribution of your data.

df.describe()

5. value_counts()

Returns the frequency of each unique value in a column. Invaluable for understanding the distribution of categorical data at a glance.

# How many countries exist per continent?
df['Continent'].value_counts()

6. corr()

Calculates the correlation between all numeric columns. Returns values from -1.0 (perfect negative relationship) to 1.0 (perfect positive relationship), with 0.0 meaning no relationship.

Important: corr() requires numeric columns only. Always use select_dtypes() before calling it to avoid a ValueError.

# Safe pattern — works on any DataFrame
df.select_dtypes(include='number').corr()

# Or select specific columns manually
df[['2022 Population', 'Area (km2)']].corr()

7. select_dtypes()

Filters and returns only the columns that match a specified data type. Useful when you need to work on a subset of columns without knowing their names in advance.

df.select_dtypes(include='number')              # numeric columns only (int and float)
df.select_dtypes(include='object')              # string columns only
df.select_dtypes(include='bool')                # boolean columns only
df.select_dtypes(include=['int64', 'float64'])  # specific numeric types
df.select_dtypes(exclude='object')              # drop all string columns
df.select_dtypes(exclude='number')              # drop all numeric columns
df.select_dtypes(include='number', exclude='bool')  # numeric but not boolean

8. Select Specific Columns

Returns a new DataFrame containing only the columns you need. Helps reduce output to what is relevant.

df[['Country', '2022 Population']]

Matplotlib Global Settings — plt.rcParams

While not a Pandas function, plt.rcParams is something you'll configure right alongside your Pandas setup. It controls the default appearance of all plots in your session — set it once at the top of your notebook and every chart inherits those settings automatically.

import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = (20, 8)   # 20 wide, 8 tall (in inches)
plt.rcParams['font.size']       = 14       # default font size for all text
plt.rcParams['figure.dpi']      = 100      # resolution (higher = sharper)
plt.rcParams['lines.linewidth'] = 2        # default line width
plt.rcParams['axes.grid']       = True     # show grid on all plots by default

Common Patterns

Filter, then Sort

The most frequent real-world pattern: narrow the dataset to a subset, then rank within that subset.

# Top 3 Asian countries by population
df[df['Continent'] == 'Asia'].nlargest(3, '2022 Population')

Sort and Preview

Both lines produce the same result — the second is preferred for its clarity.

df.sort_values('2022 Population', ascending=False).head(10)
df.nlargest(10, '2022 Population')  # ✅ preferred

Clean Display of Numbers

Control how many decimal places are shown across the entire session.

pd.set_option('display.float_format', '{:.2f}'.format)

Complete Summary Table

Function	Category	Purpose
`sort_values()`	Sorting	Sort by column values
`sort_index()`	Sorting	Sort by index
`nlargest()`	Sorting	Get top N largest values
`nsmallest()`	Sorting	Get top N smallest values
`rank()`	Sorting	Assign rank to each row
`head()`	Displaying	Show first N rows
`tail()`	Displaying	Show last N rows
`sample()`	Displaying	Show random N rows
`info()`	Displaying	Show structure and data types
`describe()`	Displaying	Statistical summary
`value_counts()`	Displaying	Count frequency of unique values
`corr()`	Displaying	Correlation between numeric columns
`select_dtypes()`	Displaying	Filter columns by data type
`plt.rcParams`	Matplotlib	Set global defaults for all plots

This is Part 1 of the Pandas for Data Science series. Next up: Part 2—GroupBy & Indexing.

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw

NumPy vs List: SIMD Vector Processing with Concrete Mathematical Example

Hussein Mahdi — Mon, 09 Feb 2026 11:39:00 +0000

SIMD, or Single Instruction Multiple Data, represents a parallel processing paradigm that exploits the vector processing capabilities inherent in modern CPU architectures. Contemporary processors contain specialized registers, typically 128-bit, 256-bit, or 512-bit in width, capable of holding multiple data elements simultaneously. When a SIMD instruction executes, it performs the same operation on all data elements within the register in a single clock cycle, rather than processing each element sequentially.

The Task

Consider the operation of adding two arrays, each containing eight floating-point numbers:

Array A: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]
Array B: [0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5]
Result:  [1.5, 3.5, 5.5, 7.5, 9.5, 11.5, 13.5, 15.5]

Traditional Python List Approach

Using standard Python lists, the computation executes sequentially:

list_a = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]
list_b = [0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5]
result = []

for i in range(len(list_a)):
    result.append(list_a[i] + list_b[i])

This approach processes one addition per iteration, requiring eight separate operations executed sequentially. Each iteration involves retrieving Python objects, extracting numerical values, performing the addition, creating a new object, and appending it to the result list.

NumPy with SIMD Acceleration

NumPy transforms this operation dramatically:

import numpy as np

array_a = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0])
array_b = np.array([0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5])
result = array_a + array_b

When executing this vectorized operation, NumPy leverages SIMD instructions. With a processor supporting 256-bit AVX instructions, the computation proceeds as follows. The CPU loads four values from each array into 256-bit vector registers simultaneously. In a single instruction cycle, it performs all four additions in parallel. The processor then loads the next four values and repeats the parallel operation. This reduces eight sequential operations to merely two parallel operations.

Note 1: Type Coercion in NumPy

What Happened?

import numpy as np

a = np.array([1, 2, 4, 5, "code"])

When you created the array with mixed integers and a string, NumPy identified that these elements could not coexist with their original data types. The system resolved this incompatibility by converting all elements to a single common type that could accommodate every value. In this case, NumPy selected the string data type (specifically, Unicode string) because strings can represent both numeric characters and alphabetic characters, whereas integers cannot represent text.

The output reveals that the first element, originally the integer 1, has been converted to the string '1'. All elements in the array are now strings, maintaining the fundamental requirement of homogeneity. The data type designation resembling <U21 indicates a Unicode string with a maximum length sufficient to store the longest element.

import numpy as np

a = np.array([1, 2, 4, 5, "code"])
print(a)  # ['1' '2' '4' '5' 'code'] <U21
print(a.dtype)  # This will show: <U21 or similar (Unicode string)
print(type(a[0]))  # This will show: <class 'numpy.str_'>

Type Coercion Hierarchy

NumPy follows a specific hierarchy when performing automatic type conversion. If an array contains integers and floating-point numbers, all integers convert to floats. If an array contains numbers and strings, all numbers convert to strings. If an array contains booleans and integers, booleans convert to integers (True becomes 1, False becomes 0). This hierarchy ensures that no data is lost during the conversion process, as the target type can always represent the source type's values.

Performance Implications

When our array contains string representations of numbers rather than actual numeric types, NumPy cannot utilize SIMD vector processing or other numerical optimizations.

Note 2: Shape Requirements in NumPy

Why the Error?

import numpy as np1

a = np1.array([[1,2,7,4], [1,2,4,6]])
print(a,"\n",a.shape,"\n",a.ndim)

NumPy requires that all dimensions of a multi-dimensional array maintain consistent sizes, creating what is termed a rectangular or uniform shape structure.

Python lists permit this irregular structure without issue, allowing nested lists of varying lengths.

Alternative Approaches for Irregular Data

a = np.array([[1, 2, 7, 4], [1, 2, 4]], dtype=object)

This approach stores references to Python list objects rather than storing numeric data directly, eliminating the possibility of vectorized operations and SIMD processing.

List Advantages and Use Cases

Python lists maintain advantages in specific contexts. Their flexibility permits storing mixed data types and facilitates dynamic resizing operations. Lists prove superior for small datasets where NumPy's initialization overhead outweighs computational benefits, for operations requiring frequent insertions or deletions at arbitrary positions, and for general-purpose programming tasks not centered on numerical computation.

Comparative Analysis: NumPy Arrays vs. Python Lists

Attribute	NumPy Arrays	Python Lists
Homogeneous data types	✓	✗
Heterogeneous data types	✗	✓
Contiguous memory storage	✓	✗
SIMD vector processing support	✓	✗
Vectorized operations	✓	✗
Low memory overhead	✓	✗
Fast numerical computations	✓	✗
Efficient element-wise operations	✓	✗
Dynamic resizing efficiency	✗	✓
Arbitrary position insertion/deletion	✗	✓
Mixed type storage capability	✗	✓
Broadcasting functionality	✓	✗
Multi-dimensional indexing	✓	✗
Built-in mathematical functions	✓	✗
Integration with scientific libraries	✓	✗
Suitable for general-purpose programming	✗	✓
Optimal for small datasets	✗	✓
Memory-efficient for large numeric data	✓	✗

Conclusion

This comparison demonstrates that NumPy and Python lists serve complementary purposes within the Python ecosystem. NumPy excels in numerical computation contexts where performance and memory efficiency matter, while Python lists provide flexibility and convenience for general-purpose programming tasks requiring heterogeneous data structures or frequent modifications.

Follow me for more content! If you enjoyed this article and want to stay updated with more technical insights, tutorials, and research on software development, AI/ML, and computer science topics, connect with me on LinkedIn and check out my projects on GitHub. I regularly share valuable content about .NET development, algorithms, machine learning, and software architecture. Let's learn and grow together!

State Pattern: Transforming Objects Through Internal States

Hussein Mahdi — Fri, 09 May 2025 09:47:00 +0000

In software development, we often encounter objects whose behavior must change based on their internal state. Think of a document (draft, under review, published), an order (new, paid, shipped, delivered), or as we'll explore, a student moving through academic stages.

However, with the advancement of programming and the complexity of the problems we face, we discovered that every object has two types of attributes: static attributes and state attributes , which will be the focus of our discussion now. Because these attributes have behavior and state rather than a fixed value, dealing with them can be complex. Changing an object's state may result in a change in its behavior. For example, a person's behavior when they were children differs from their behavior when they became adolescents, even though they remain the same "human."

Therefore, it was found preferable to isolate these attributes in an abstract interface. Within the abstract interface, there are functions known as state functions which cause states to change and are similar to rules. In some cases, they are called state operator functions Each state then executes these functions, but according to the implementation appropriate for that state.

When handled improperly, state-dependent behavior leads to complex, difficult-to-maintain code filled with conditional statements. The State Pattern offers an elegant solution to this problem.

What is the State Pattern?

The State Pattern allows an object to change its behavior when its internal state changes. The pattern makes the object appear to change its class by encapsulating state-specific behavior in separate state classes and eliminating complex conditional statements.

As defined by the Gang of Four, this pattern:

Encapsulates state-specific behavior in separate classes
Delegates state-specific behavior to the current state object
Allows for easy addition of new states with minimal changes to existing code

The Problem: Evolving Student States

Let's consider a concrete example: a Student class that handles different behaviors as the student progresses through exam preparation:

Preparing: Student is studying for the exam
ExamTaken: Student has taken the exam and is awaiting results
Passed: Student has passed and completed the course

Traditional Approach with Conditionals

Here's how this might be implemented traditionally:

public class Student {
    // State constants
    public static final int PREPARING = 0;
    public static final int EXAM_TAKEN = 1;
    public static final int PASSED = 2;

    private int state;
    private String name;

    public Student(String name) {
        this.name = name;
        // Initial state is PREPARING
        this.state = PREPARING;
    }

    public void study() {
        System.out.println(name + " is attempting to study.");

        if (state == PREPARING) {
            System.out.println("Studying harder to be prepared for the exam!");
        } else if (state == EXAM_TAKEN) {
            System.out.println("Already took the exam! Waiting for results...");
        } else if (state == PASSED) {
            System.out.println("Studying for the next course!");
        } else {
            System.out.println("Unknown state!");
        }
    }

    public void takeExam() {
        System.out.println(name + " is attempting to take the exam.");

        if (state == PREPARING) {
            System.out.println("Taking the exam now...");
            state = EXAM_TAKEN;
        } else if (state == EXAM_TAKEN) {
            System.out.println("Cannot take the exam again! Already taken.");
        } else if (state == PASSED) {
            System.out.println("Already passed this exam! Moving to next course.");
        } else {
            System.out.println("Unknown state!");
        }
    }

    public void receiveResults(boolean passed) {
        System.out.println(name + " is receiving exam results.");

        if (state == PREPARING) {
            System.out.println("Cannot receive results yet - haven't taken the exam!");
        } else if (state == EXAM_TAKEN) {
            System.out.println("Receiving exam results...");
            if (passed) {
                state = PASSED;
                System.out.println("Passed the exam!");
            } else {
                state = PREPARING;
                System.out.println("Failed the exam. Need to prepare again!");
            }
        } else if (state == PASSED) {
            System.out.println("Already received the results. Passed!");
        } else {
            System.out.println("Unknown state!");
        }
    }

    public String getStateName() {
        if (state == PREPARING) {
            return "Preparing for exam";
        } else if (state == EXAM_TAKEN) {
            return "Waiting for results";
        } else if (state == PASSED) {
            return "Passed the exam";
        } else {
            return "Unknown";
        }
    }

    @Override
    public String toString() {
        return name + " - Current state: " + getStateName();
    }
}

Problems with the Traditional Approach

This implementation has several issues:

Code Duplication: The same state checks are repeated in every method
Complex Conditionals: Each method contains multiple if-else branches
Scattered State Transitions: State changes are dispersed throughout the code
Poor Maintainability: Adding a new state requires modifying every method
Error Prone: Easy to miss updating a condition when adding states
Hard to Read: Understanding all possible state transitions is difficult
Violation of OCP: The class must be modified to add new states
Violation of SRP: The Student class is responsible for all state behaviors

The Solution: Applying the State Pattern

Instead of managing state transitions with conditionals, we can use the State Pattern to separate each state's behavior into its own class:

1. Define the State Interface

interface StudentState {
    void study(Student student);
    void takeExam(Student student);
    void receiveResults(Student student, boolean passed);
    String getStateName();
}

2. Implement Concrete States

class PreparingState implements StudentState {
    @Override
    public void study(Student student) {
        System.out.println("Studying harder to be prepared for the exam!");
    }

    @Override
    public void takeExam(Student student) {
        System.out.println("Taking the exam now...");
        student.setState(new ExamTakenState());
    }

    @Override
    public void receiveResults(Student student, boolean passed) {
        System.out.println("Cannot receive results yet - haven't taken the exam!");
    }

    @Override
    public String getStateName() {
        return "Preparing for exam";
    }
}

class ExamTakenState implements StudentState {
    @Override
    public void study(Student student) {
        System.out.println("Already took the exam! Waiting for results...");
    }

    @Override
    public void takeExam(Student student) {
        System.out.println("Cannot take the exam again! Already taken.");
    }

    @Override
    public void receiveResults(Student student, boolean passed) {
        System.out.println("Receiving exam results...");
        if (passed) {
            student.setState(new PassedState());
            System.out.println("Passed the exam!");
        } else {
            student.setState(new PreparingState());
            System.out.println("Failed the exam. Need to prepare again!");
        }
    }

    @Override
    public String getStateName() {
        return "Waiting for results";
    }
}

class PassedState implements StudentState {
    @Override
    public void study(Student student) {
        System.out.println("Studying for the next course!");
    }

    @Override
    public void takeExam(Student student) {
        System.out.println("Already passed this exam! Moving to next course.");
    }

    @Override
    public void receiveResults(Student student, boolean passed) {
        System.out.println("Already received the results. Passed!");
    }

    @Override
    public String getStateName() {
        return "Passed the exam";
    }
}

3. Refactor the Context Class

class Student {
    private StudentState state;
    private String name;

    public Student(String name) {
        this.name = name;
        // Initial state is Preparing
        this.state = new PreparingState();
    }

    public void setState(StudentState state) {
        this.state = state;
    }

    public void study() {
        System.out.println(name + " is attempting to study.");
        state.study(this);
    }

    public void takeExam() {
        System.out.println(name + " is attempting to take the exam.");
        state.takeExam(this);
    }

    public void receiveResults(boolean passed) {
        System.out.println(name + " is receiving exam results.");
        state.receiveResults(this, passed);
    }

    public String getStateName() {
        return state.getStateName();
    }

    @Override
    public String toString() {
        return name + " - Current state: " + state.getStateName();
    }
}

Benefits of the State Pattern

Clean Code: Eliminates messy conditionals
Maintainability: State-specific code is localized
Extensibility: New states can be added easily
Flexibility: States can be swapped at runtime
Testability: Each state can be tested in isolation
Readability: State transitions are explicit
Single Responsibility: Each state class has one purpose
Open-Closed Principle: Student class is closed for modification but open for extension

State Pattern vs. Strategy Pattern: Key Differences

The State and Strategy patterns share a similar structure but have different intents:

Aspect	Strategy Pattern	State Pattern
Intent	Defines a family of interchangeable algorithms	Changes object behavior when internal state changes
Control	Client code selects strategy	States often control transitions to other states
Relationship	Strategies typically independent	States often know about and reference each other
Example	Client decides duck's flying behavior	Student automatically changes own state

When to Use the State Pattern

The State Pattern is particularly useful when:

An object's behavior depends on its state
You have multiple operations with state-dependent behavior
State transitions follow well-defined rules
You need to eliminate complex conditional logic

It might be overkill when:

You have few states or simple transitions
States rarely change
State-specific behavior is minimal

Real-World Examples

The State Pattern appears in many modern frameworks and libraries:

XState: JavaScript library implementing formal state machines to manage complex UI state transitions
RxJava: Uses Observer pattern primarily but incorporates state management for asynchronous stream processing
Unidraw: Framework for graphical editors that employs state-like behavior for tool switching

Conclusion

The State Pattern is a powerful tool for managing complex state-dependent behavior. By encapsulating each state in its own class, you create code that's easier to understand, test, and extend. Next time you find yourself writing a growing number of state-checking conditionals, consider if the State Pattern might provide a more elegant solution.

This article is inspired by concepts from "Head First Design Patterns" by Eric Freeman & Elisabeth Robson and "Design Patterns: Elements of Reusable Object-Oriented Software" by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides.

Analyzing YOLO Architecture: Part 2 - The Neck Component

Hussein Mahdi — Wed, 30 Apr 2025 08:43:00 +0000

The Neck Component in YOLO Architecture: Feature Aggregation and Multi-Scale Fusion

I'm continuing my deep dive into YOLO architecture components with a new paper that examines the often overlooked but critically important "neck" component.

What's the neck component?

The neck serves as the bridge between the backbone network (feature extractor) and the head (detection component), performing critical functions including:

Feature fusion across different scales
Enhancement of information flow between layers
Feature refinement
Balancing of resolution and semantic information

Evolution across YOLO versions

My analysis traces the development from:

YOLOv1's lack of a dedicated neck component
YOLOv2's introduction of the passthrough layer
YOLOv3's adoption of Feature Pyramid Networks
YOLOv4 and beyond implementing advanced architectures like PANet

Series progression

This paper is the second in my YOLO architecture series:

I'm working through each major component to provide a comprehensive understanding of modern object detection architectures.

Discussion

I'd love to hear your thoughts and experiences with YOLO architectures! Have you implemented custom necks or experimented with different feature fusion techniques? What improvements have you seen?

Stay tuned for Part 3 where I'll analyze the detection head component!

The Proxy Pattern: A Software Developer’s Essential Tool

Hussein Mahdi — Fri, 25 Apr 2025 13:24:04 +0000

The Proxy Pattern is a structural design pattern that provides a surrogate or placeholder for another object to control access to it. It's like having a representative that acts on behalf of another object.

The Proxy Pattern allows you to:
• Provide a substitute or placeholder for another object

• Control access to the original object

• Perform operations before or after requests reach the original object

When to Use the Proxy Pattern

Lazy initialization is needed: When creating objects is expensive and you want to delay creation until absolutely necessary
Access control is required: When you need to restrict access to certain operations or resources based on permissions
Resource management is important: When working with limited resources that need controlled allocation
Added functionality is needed before/after access: When you want to perform operations before or after accessing the real object (like logging or caching)

The key indicator is when you need to control access to an object without changing its interface.

Structure of the Proxy Pattern
The pattern consists of these key components:
• Subject: An interface that defines the common operations for both the RealSubject and Proxy

• RealSubject: The actual object that the proxy represents

• Proxy: Maintains a reference to the RealSubject and controls access to it

• Client: Interacts with the Subject interface

Types of Proxies
1. Virtual Proxy
• Creates expensive objects on demand (lazy initialization)
• Example: Loading large images in a document only when they need to be displayed

2. Protection Proxy
• Controls access permissions to objects
• Example: Access control systems that verify user permissions before allowing resource access

3. Remote Proxy
• Represents objects in different address spaces
• Example: A local object representing a remote service

4. Smart Reference
• Performs additional actions when objects are accessed
• Example: Reference counting, locking mechanisms

Example: Virtual Proxy for Image Loading in C#

Problem: Loading high-resolution images is expensive and may not be immediately necessary.

Solution: Use a Virtual Proxy to load the actual image only when it needs to be displayed.

Benefits of the Proxy Pattern

Improved performance: Defers costly object creation until necessary
Enhanced security: Controls access to sensitive objects
Reduced complexity: Hides implementation details from clients
Increased flexibility: Enables features like caching, logging, and access control without changing the original object
Location transparency: Provides unified access to distributed objects

Limitations of the Proxy Pattern

Increased response time: Adds a level of indirection
Implementation complexity: Some proxy types can be complex to implement
Potential for overuse: Not every object needs a proxy

Real-World Applications

Web browsers: Caching proxy for web pages
ORM frameworks: Proxies for lazy loading of database entities
Credit cards: Act as proxies for bank accounts

Summary
The Proxy Pattern provides a surrogate for another object to control access to it. It's particularly useful for lazy loading, access control, and adding behavior when accessing objects. The pattern maintains the same interface as the original object while providing additional functionality.

Reference : Book : ”Design Patterns: Elements of Reusable Object-Oriented Software” by Gamma, Helm, Johnson, and Vlissides. also call Gun of Fure

The Facade Pattern: Simplifying Complex Systems

Hussein Mahdi — Thu, 10 Apr 2025 08:09:00 +0000

In modern software development, we often work with complex systems that consist of numerous interrelated components. These systems, while powerful, can be challenging to understand and use effectively. The Facade Pattern offers an elegant solution to this problem by providing a simplified interface to a complex subsystem.

The Problem: Complexity in Action

Consider a home theater system with multiple components:

Popcorn popper
Theater lights
Amplifier
Projection screen
Projector
CD player
Tuner
DVD player

To watch a movie, a user would need to perform 13 separate operations:

Turn on the popcorn popper
Start the popper popping
Dim the lights
Put the screen down
Turn the projector on
Put the projector on wide-screen mode
Set the projector input to DVD
Turn the sound amplifier on
Set the amplifier to DVD input
Set the amplifier volume to medium
Turn the DVD player on
Set the amplifier to surround sound
Start the DVD player playing

This process is cumbersome, error-prone, and requires detailed knowledge of each component and their interactions. Clearly, we need a highly abstract approach that simplifies our interaction with the system without worrying about the details.

The Facade Pattern: Definition and Purpose

The Facade Pattern provides a unified interface to a set of interfaces in a subsystem. It defines a higher-level interface that makes the subsystem easier to use by hiding its complexity from clients.

The pattern emerged from the need to simplify interactions with complex systems. As software systems grow, they often develop intricate subsystems with numerous components and relationships. This complexity creates several challenges:

Steep learning curves for new developers
Increased coupling between clients and implementation details
Difficulty in maintaining code that interacts with many components
Error-prone integration processes

How It Works

The Facade Pattern works by:

Creating a new class (the facade) that presents a simplified interface
Internally, the facade manages the complex interactions with subsystem components
Clients interact only with the facade rather than directly with the subsystem
The facade translates simple, high-level requests into the appropriate sequence of subsystem operations

This approach shields clients from the complexity of the subsystem while still allowing them to access its functionality.

Basic Components

The Facade Pattern consists of:

The Facade: A class that provides a simplified interface to the subsystem
The Subsystem: A set of complex classes that implement functionality
The Client: Code that interacts with the subsystem through the façade

Important Note: The pattern does not alter the subsystem classes or prevent clients from accessing them directly if needed. It simply offers an easier way to use the subsystem for common tasks.

Solution to the Home Theater Problem

To solve our home theater complexity problem, we can create a HomeTheaterFacade class that wraps all the subsystem components and provides simplified methods:

public class HomeTheaterFacade {

    // Component references
    Amplifier amp;
    Tuner tuner;
    DvdPlayer dvd;
    CdPlayer cd;
    Projector projector;
    TheaterLights lights;
    Screen screen;
    PopcornPopper popper;

    // Constructor with component initialization
    public void watchMovie(String movie) {

        System.out.println("Get ready to watch a movie...");
        popper.on();
        popper.pop();
        lights.dim(10);
        screen.down();
        projector.on();
        projector.wideScreenMode();
        amp.on();
        amp.setDvd(dvd);
        amp.setSurroundSound();
        amp.setVolume(5);
        dvd.on();
        dvd.play(movie);
    }

    public void endMovie() {
        // Similar sequence to shut everything down
    }

    // Other methods for different use cases
}

Now, instead of 13 separate operations, the client simply calls:

homeTheater.watchMovie("Raiders of the Lost Ark");
// When finished
homeTheater.endMovie();

The Principle of Least Knowledge

The Facade Pattern supports the Principle of Least Knowledge (or Law of Demeter), which states: "Talk only to your immediate friends." This principle guides us to reduce the interactions between objects to just a few closely related classes.

For any object, you should only invoke methods that belong to:

The object itself
Objects passed as parameters to a method
Any object the method creates
Any direct component objects

The Facade Pattern supports this principle by:

Providing clients with a single point of contact (the facade)
Handling all the complex interactions with the subsystem internally
Reducing the number of objects the client needs to know about
Creating clear boundaries between the client and the subsystem

This reduces dependencies and makes the system more maintainable.

Benefits of the Facade Pattern

Simplified Interface: Provides a simple interface to a complex subsystem, making it easier to use.
Reduced Coupling: Decreases the dependency between client code and subsystem components.
Subsystem Flexibility: Allows the subsystem to change without affecting client code.
Improved Code Readability: Makes client code more readable and focused on high-level tasks.
Better System Organization: Creates clear layers in your architecture.

Disadvantages of the Facade Pattern

Limited Functionality Access: May restrict access to some advanced features of the subsystem.
Additional Indirection Layer: Adds an extra layer in the architecture.
Potential Facade Bloat: The facade class can become too large if not designed carefully.
Maintenance Overhead: Requires updating when the subsystem changes.
Minor Performance Impact: May introduce slight overhead due to delegation.

Relation with Other Patterns

The Facade Pattern differs from similar patterns in key ways:

Facade vs. Adapter

Purpose: Adapter converts one interface to another; Facade simplifies complex interfaces
Interface Change: Adapter changes interface format; Facade maintains format but simplifies
Intent: Adapter makes incompatible interfaces compatible; Facade hides complexity

Facade vs. Abstract Factory

Purpose: Abstract Factory creates families of related objects; Facade simplifies access to subsystem functionality
Intent: Abstract Factory abstracts object creation; Facade abstracts subsystem use
Client Knowledge: Abstract Factory client creates objects; Facade client uses existing objects

Application in Layered Architecture

The Facade Pattern is frequently used in layered architectures where it acts as a gateway between layers. For example, in a typical application with UI, Domain, and Data layers, facades can provide clean interfaces between these layers.

Modern Applications: CQRS Facades

In modern software architecture, the Facade Pattern finds application in Command Query Responsibility Segregation (CQRS) where separate facades are used for commands (write operations) and queries (read operations), providing cleaner code separation.

CQRS Facades: Cleaner Code Separation | by Anthony Miller | Medium

Conclusion

The Facade Pattern offers a powerful way to manage complexity in software systems. By providing a simplified interface to complex subsystems, it makes code more maintainable, readable, and flexible. While it does introduce an additional layer of abstraction, the benefits in terms of usability and maintainability often outweigh this cost, particularly in systems with complex components that need to work together seamlessly.

When designing your next complex system, consider whether a facade might help simplify your client code and create cleaner boundaries between system components.

Binary Tree All Operations as pseudocode

Hussein Mahdi — Sat, 05 Apr 2025 08:11:00 +0000

Tree Structure

Each node x has these attributes:

x.key: key stored in the node
x.p: pointer to the parent node
x.left: pointer to the left child
x.right: pointer to the right child

A tree T has:

T.root: pointer to the root node

Algorithms

1. Inorder Tree Walk

Input: x (pointer to a node in the binary search tree)
Output: None (prints keys in order)

1. If x ≠ null
2.   Inorder-tree-walk(x.left)
3.   Print x.key
4.   Inorder-tree-walk(x.right)

2. Tree Search (Recursive)

Input: x (pointer to subtree root), k (key to search)
Output: Pointer to node with key k (or null if not found)

1. If x == null or k == x.key
2.   Return x
3. If k < x.key
4.   Return tree-search(x.left, k)
5. Else 
6.   Return tree-search(x.right, k)

3. Iterative Tree Search

Input: x (pointer to subtree root), k (key to search)
Output: Pointer to node with key k (or null if not found)

1. While x ≠ null and k ≠ x.key
2.   If k < x.key
3.     x = x.left
4.   Else 
5.     x = x.right
6. Return x

4. Tree Minimum

Input: x (pointer to non-null node)
Output: Pointer to node with minimum key in the subtree

1. While x.left ≠ null
2.   x = x.left
3. Return x

5. Tree Maximum

Input: x (pointer to non-null node)
Output: Pointer to node with maximum key in the subtree

1. While x.right ≠ null
2.   x = x.right
3. Return x

6. Tree Insert

Input: t (binary search tree), z (node to insert)
Output: None (modifies tree by inserting z)

1. y = null
2. x = t.root
3. While x ≠ null
4.   y = x
5.   If z.key < x.key
6.     x = x.left
7.   Else 
8.     x = x.right
9. z.p = y
10. If y == null
11.   t.root = z
12. Elseif z.key < y.key
13.   y.left = z
14. Else 
15.   y.right = z

7. Transplant (Helper for Delete)

Input: t (binary search tree), u (subtree to replace), v (replacing subtree)
Output: None (modifies tree)

1. If u.p == null
2.   t.root = v
3. Elseif u == u.p.left
4.   u.p.left = v
5. Else 
6.   u.p.right = v
7. If v ≠ null
8.   v.p = u.p

8. Tree Delete

Input: t (binary search tree), z (node to delete)
Output: None (modifies tree)

1. If z.left == null
2.   Transplant(t, z, z.right)
3. Elseif z.right == null
4.   Transplant(t, z, z.left)
5. Else
6.   y = tree-minimum(z.right)
7.   If y.p ≠ z
8.     Transplant(t, y, y.right)
9.     y.right = z.right
10.    y.right.p = y
11.    Transplant(t, z, y)
12.    y.left = z.left
13.    y.left.p = y

Additional Tree Operations

Size

Input: T (binary tree)
Output: Number of nodes in the tree

1. If T.root == Null
2.   Return 0
3. count = 0
4. sizeHelper(T.root, count)
5. Return count

Other Basic Operations

isEmpty(): Returns true if tree is empty
root(): Returns the root node
parent(v): Returns parent of node v
children(v): Returns list of children of node v
leftChild(v): Returns left child of node v
rightChild(v): Returns right child of node v
sibling(v): Returns sibling of node v

Traversal Algorithms

Preorder Traversal

Input: v (node)
Output: None

1. If v == Null
2.   Return
3. Visit v
4. If isInternal(v)
5.   preorder(v.left)
6.   preorder(v.right)

Postorder Traversal

Input: v (node)
Output: None

1. If v == Null
2.   Return
3. If isInternal(v)
4.   postorder(v.left)
5.   postorder(v.right)
6. Visit v

Inorder Traversal

Input: v (node)
Output: None

1. If v == Null
2.   Return
3. If v.left ≠ Null
4.   inorder(v.left)
5. Visit v
6. If v.right ≠ Null
7.   inorder(v.right)

Euler Tour Traversal

Input: v (node)
Output: None

1. If v == null
2.   Return
3. visitBeforeSubtrees(v)
4. If isInternal(v)
5.   If v.left ≠ Null
6.     eulerTour(v.left)
7.   visitBetweenSubtrees(v)
8.   If v.right ≠ Null
9.     eulerTour(v.right)
10. visitAfterSubtrees(v)

Time Complexity Summary

Operation	Time Complexity
Inorder Walk	O(n)
Tree Search	O(h)
Tree Insert	O(h)
Tree Delete	O(h)
Size	O(n)
Traversals	O(n)

Note: h = height of the tree, n = number of nodes

Euler Tour Example

The Euler Tour captures all tree traversal information in a single walk:

Each node is visited exactly 3 times
Combines preorder, inorder, and postorder traversals

Example Tree Expression

The Euler Tour can help evaluate complex tree expressions:

((3+1)×(9−5))−(3×(7−4)+6)

This demonstrates how Euler Tour can be used to systematically process and evaluate tree-based mathematical expressions.

Binary Search Tree Operations Complexity Table

Operation	Time Complexity (Big-O)
inorder-tree-walk	O(n)
tree-search	O(h)
iterative-tree-search	O(h)
tree-minimum	O(h)
tree-maximum	O(h)
tree-insert	O(h)
tree-delete	O(h)
transplant	O(1)
size	O(n)
isEmpty	O(1)
root	O(1)
parent	O(1)
children	O(1)
leftChild	O(1)
rightChild	O(1)
sibling	O(1)
isInternal	O(1)
isExternal	O(1)
isRoot	O(1)
preorder	O(n)
postorder	O(n)
inorder	O(n)
eulerTour	O(n)

Euler Tour Example with Larger Tree

Euler Tour traversal sequence for a tree with 3 layers:

A (first visit)
B (first visit)
D (first visit)
H (first visit)
H (second visit) - no children
H (third visit)
D (second visit)
I (first visit)
I (second visit) - no children
I (third visit)
D (third visit)
B (second visit)
E (first visit)
J (first visit)
J (second visit) - no children
J (third visit)
E (second visit) - E has no right child
E (third visit)
B (third visit)
A (second visit)
C (first visit)
F (first visit)
F (second visit) - no children
F (third visit)
C (second visit)
G (first visit)
G (second visit) - no children
G (third visit)
C (third visit)
A (third visit)

Note: Each node is visited exactly 3 times, creating a complete "tour" around the tree that captures all the information from preorder, inorder, and postorder traversals in a single walk.

Mathematical Expression Using Euler Tour Algorithm

Q: What is the mathematical expression for this tree when using the Euler Tour algorithm?

Solution:

Start with the left subtree of the root (-):
- Left subtree of ×: (3 + 1)
- Right subtree of ×: (9 - 5)
- Combine: (3 + 1) × (9 - 5)
Move to the right subtree of the root (-):
- Left subtree of +: 3 × (7 - 4)
- Right subtree of +: 6
- Combine: 3 × (7 - 4) + 6
Combine the results from the left and right subtrees of the root:
- Left: (3 + 1) × (9 - 5)
- Right: 3 × (7 - 4) + 6
- Combine: ((3 + 1) × (9 - 5)) - (3 × (7 - 4) + 6)

Final Answer:
The mathematical expression represented by the tree is:

((3+1)×(9−5))−(3×(7−4)+6) => ((4)×(4))−(3×(3)+6) = 16 − (3×3 + 6) = 16 − (9 + 6) = 16 − 15 = 1

Data Structure Operations Time Complexity Guide

Hussein Mahdi — Sat, 15 Mar 2025 16:19:00 +0000

Understanding time complexity is essential for writing efficient code. Here's a quick reference guide for common data structure operations to help you make the right choices in your projects.

Index-Based List Operations

Operation	Time Complexity	Description
get(r)	O(1)	Retrieve element at index r
set(r, e)	O(1)	Replace element at index r with e
add(r, e)	O(n)	Insert element e at index r
remove(r)	O(n)	Remove element at index r

Linked Lists (Position-Based) Operations

Operation	Time Complexity	Description
element()	O(1)	Return element at a given position
first()	O(1)	Return position of first element
last()	O(1)	Return position of last element
before(p)	O(1)	Return position before p
after(p)	O(1)	Return position after p
insertAfter(p, e)	O(1)	Insert element e after position p
insertBefore(p, e)	O(1)	Insert element e before position p
remove(p)	O(1)	Remove element at position p

Tree Operations

Operation	Time Complexity	Description
createNode(element, parent)	O(1)	Create new tree node
root()	O(1)	Return root node of tree
parent(v)	O(1)	Return parent of node v
children(v)	O(1)	Return set of children of node v
isInternal(v)	O(1)	Check if node v is internal
isExternal(v)	O(1)	Check if node v is external (leaf)
isRoot(v)	O(1)	Check if node v is the root
size()	O(1)	Return number of nodes in tree
elements()	O(n)	Return set of all elements in tree
positions()	O(n)	Return set of all positions in tree
swapElements(v, w)	O(1)	Swap elements between nodes v and w
replaceElement(v, e)	O(1)	Replace element in node v with e
depth(T, v)	O(n)	Calculate depth of node v in tree T
height(T, v)	O(n)	Calculate height of subtree rooted at v

Tree Traversal Algorithms

Operation	Time Complexity	Description
preorder(T, v)	O(n)	Visit nodes in preorder starting from v
postorder(T, v)	O(n)	Visit nodes in postorder starting from v

Why This Matters

Understanding the time complexity of these operations helps you:

Choose the right data structure for your specific use case
Optimize your algorithms for better performance
Avoid unexpected performance bottlenecks in production

Keep this reference handy when designing your next algorithm or debugging performance issues!

Implementation Examples
If you're looking for pseudocode implementations of these data structures, check out my GitHub repository:
Algorithms Library on GitHub

The repository contains pseudocode implementations that complement this time complexity guide. Feel free to use them as a reference when implementing these data structures in your projects.

Code Smells Between Classes: The Second Chapter of Modern Software Design

Hussein Mahdi — Sun, 23 Feb 2025 10:15:00 +0000

In our previous discussion on code smells (which you can find Code Smells: Understanding Design Evolution in Modern Software Development), we explored how these design weaknesses manifest within individual classes. Now, let's dive into an equally crucial aspect - code smells that occur between classes. These inter-class relationships often reveal deeper architectural challenges that can significantly impact system maintainability and scalability.

Just as we saw with intra-class code smells, the interactions between classes can either strengthen or weaken our software's architecture. While our previous discussion focused on issues like long methods and large classes, here we'll examine how classes interact with each other and the common pitfalls that emerge from these relationships. Let's explore these patterns with practical examples in C#.

Feature Envy
This smell occurs when a method in one class seems more interested in another class than its own. Here's what it looks like:

Inappropriate Intimacy
When two classes are too tightly coupled, knowing too much about each other's private details:

Message Chains
When you have to navigate through multiple objects to get to the desired data:

Shotgun Surgery
When making a change requires updating many different classes:

Middle Man
When a class performs no real function except delegating to another class:

Conclusion
Understanding and addressing code smells between classes is crucial for maintaining a healthy software architecture. These patterns often indicate violations of key design principles like encapsulation, the Law of Demeter, and single responsibility. By recognizing and refactoring these smells, we can create more maintainable and flexible systems that better withstand the test of time and changing requirements.

Remember 🎯, the goal isn't to eliminate every possible code smell; sometimes what appears to be a smell might be the most practical solution for your specific context. The key is to recognize these patterns and make informed decisions about when and how to address them.

Forem: Hussein Mahdi

Mastering Pandas — Part 4: Data Visualization with Matplotlib & Seaborn

From Clean Data to Clear Insight

Part 1 — Matplotlib: The Engine

What is Matplotlib?

Global Settings with plt.rcParams

Setting a Visual Theme with plt.style.use()

Plotting from a DataFrame with .plot()

Plotting Directly with plt.plot()

Decorating Any Chart

Multiple Subplots with Object-Oriented Style

Part 2 — Seaborn: Statistical Visualization

What is Seaborn?

Global Styling with set_theme()

Relational Charts — Showing Relationships Between Numbers

Distribution Charts — Showing How Values Are Spread

Categorical Charts — Comparing Groups

Matrix & Regression Charts — Correlations and Trends

Combining Seaborn with Matplotlib

Choosing the Right Tool

Complete Summary Table

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Real Data Is Never Clean

Step 1 — Inspect Before You Touch Anything

Step 2 — Fix Column Names First

Step 3 — Handle Missing Values

Step 4 — Remove Duplicates

Step 5 — Fix Data Types

Step 6 — Clean Strings

Step 7 — Replace and Standardize Values

Step 8 — Handle Outliers

Step 9 — Drop Rows and Columns

Step 10 — Create New Columns

Step 11 — Add New Rows

Step 12 — Reset the Index

Putting It All Together — A Real Cleaning Pipeline

Now Combine — Merging & Joining DataFrames

1️⃣ merge() — The Main Tool

The how= Parameter — All Four Types

The on= Parameter — What to Match On

Handling Shared Column Names — _x and _y

Other Useful Parameters

2️⃣ concat() — Stacking DataFrames

axis= Parameter

join= Parameter

ignore_index= Parameter

keys= Parameter

verify_integrity= Parameter

merge() vs concat() — When to Use Which

3️⃣ join() — Index-Based Joining

Why merge() Is Almost Always Better

Common Patterns

Clean first, then merge

Merge and immediately filter

Combine monthly files with concat, then analyze

Complete Summary Table

Mastering Pandas — Part 2: GroupBy & Indexing

From Loading to Understanding

Part 1 — GroupBy

1️⃣ Aggregation Functions — The Core of GroupBy

2️⃣ agg() — Multiple Functions at Once

3️⃣ transform() — Keep the Original Shape

4️⃣ apply() — Custom Logic Per Group

5️⃣ filter() — Keep or Drop Entire Groups

6️⃣ size() vs count() — A Critical Distinction

7️⃣ nunique() — Count Distinct Values Per Group

8️⃣ first() & last() — Boundary Rows of Each Group

9️⃣ Group by Multiple Columns

GroupBy Summary Table

Part 2 — Indexing

1️⃣ [] — Basic Indexing

2️⃣ loc[] — Label-Based Indexing

3️⃣ iloc[] — Position-Based Indexing

loc[] vs iloc[] — The Key Distinction

4️⃣ at[] & iat[] — Fast Single Value Access

5️⃣ Boolean Indexing — Filter by Condition

6️⃣ isin() — Match Against a List of Values

7️⃣ query() — Filter Using a String Expression

8️⃣ where() — Conditional Value Replacement

9️⃣ set_index() — Use a Column as the Index

Global Settings with `plt.rcParams`

Setting a Visual Theme with `plt.style.use()`

Plotting from a DataFrame with `.plot()`

Plotting Directly with `plt.plot()`

Global Styling with `set_theme()`

1️⃣ `merge()` — The Main Tool

The `how=` Parameter — All Four Types

The `on=` Parameter — What to Match On

Handling Shared Column Names — `_x` and `_y`

2️⃣ `concat()` — Stacking DataFrames

`axis=` Parameter

`join=` Parameter

`ignore_index=` Parameter

`keys=` Parameter

`verify_integrity=` Parameter

`merge()` vs `concat()` — When to Use Which

3️⃣ `join()` — Index-Based Joining

Why `merge()` Is Almost Always Better

2️⃣ `agg()` — Multiple Functions at Once

3️⃣ `transform()` — Keep the Original Shape

4️⃣ `apply()` — Custom Logic Per Group

5️⃣ `filter()` — Keep or Drop Entire Groups

6️⃣ `size()` vs `count()` — A Critical Distinction

7️⃣ `nunique()` — Count Distinct Values Per Group

8️⃣ `first()` & `last()` — Boundary Rows of Each Group

1️⃣ `[]` — Basic Indexing

2️⃣ `loc[]` — Label-Based Indexing

3️⃣ `iloc[]` — Position-Based Indexing

`loc[]` vs `iloc[]` — The Key Distinction

4️⃣ `at[]` & `iat[]` — Fast Single Value Access

6️⃣ `isin()` — Match Against a List of Values

7️⃣ `query()` — Filter Using a String Expression

8️⃣ `where()` — Conditional Value Replacement

9️⃣ `set_index()` — Use a Column as the Index

🔟 `xs()` — Cross-Section for MultiIndex

Combine GroupBy with `agg()` and `query()`