Forem: Ife

How Gradient Descent Powers Machine Learning Models

Ife — Fri, 22 Nov 2024 09:15:44 +0000

Introduction

Building accurate machine learning models relies heavily on optimization techniques and gradient descent is one of the techniques. Gradient descent helps models adjust parameters, minimise errors and improve performance overtime. In this article, I will dive into gradient descent as a concept and why it is important in the machine learning process.

What is Gradient Descent

Gradient descent is the process that minimises a model's errors by adjusting its parameters continuously until it finds the best value that reduces the loss function.

The loss function is the difference between the predicted value and the actual value. To get the predicted value, the model runs some calculation which involves parameters. These parameters determine how the model processes input data to generate its predictions, and they are adjusted during training to minimize the loss function and improve accuracy. Gradient descent handles the adjustments of these parameters.

How Gradient Descent Works

This is the equation behind the simple linear regression model. It is similar to the equation of a line.

The parameters in this equation are w and b, the weight and the bias. When the linear regression model makes a prediction using this equation, the predicted output is compared with the actual output using the following equation,

When you substitute the equation for the predicted value into the equation, you get this,

This loss function is called the Mean Squared Error (MSE). The smaller the difference between the predicted value and the actual value, the more accurate the model's predictions and this accuracy depends on the values of the weight (w) and the bias (b).

If you are to build a linear regression model that predicts house prices based on size only, the features (X) will be the size of the house and the target (y) will be the price of the house.

x_train = np.array([500, 800, 1000, 1500, 2000])
y_train = np.array([50, 80, 100, 150, 200])

If w is 0 and b is 0, the model predicts the target value to be 0:

w = 0
b = 0

pred_y = w*x_train[0] + b
# x_train[0] = 500

# pred_y = 0

The actual target value when the size of the house is 500, is 50. The loss function for this prediction is 50, which means that the model is far from accurate. However, if w and b were different, the model would also predict a different target value.

To see in real-time, the effect of gradient descent on the loss function, look at the graph below:

In the above graph, you can see that the loss reduces until it gets to a point (where w = 0.1). This minimum point is called the global minimum and the value of w at this point will give the smallest possible loss function value.

Instead of guessing the values of w and b, the gradient descent algorithm will go through all the possible values for w and b until it gets the value for w and b, resulting in the smallest possible loss function value. The gradient descent is defined as:

The gradient descent algorithm updates the w and b parameters simultaneously after each iteration using the equations above until it gets the best values that result in the best loss function value for accurate predictions.

The Role of the Learning Rate

The alpha sign in the gradient descent equations is known as the learning rate. The learning rate is a value that determines how much or how little the parameters get updated after each iteration.

If the learning rate is too small, the gradient descent will take too long to reach the global minimum. However, if the learning rate is too big, the gradient descent might miss the global minimum and that will lead to increasing values of the loss function which you don't want.

You provide the gradient descent algorithm with a good learning rate value. A good range is from 0.01 to 1.

To learn more about gradient descent and learning rate including graphs, check out this notebook on Gradient Descent.

Types of Gradient Descent

Batch Gradient Descent
This type of gradient descent computes the gradient of the entire dataset to update parameters. Each iteration uses all training examples to calculate the gradient. It is best for small datasets and when computing power is sufficient.

Stochastic Gradient Descent (SGD)
The Stochastic Gradient Descent (SGD) algorithm updates parameters for each individual training example. It iterates through examples one at a time. It is best for very large datasets or when computational efficiency is a priority.

Mini-batch Gradient Descent
This type of gradient descent combines the strengths of batch and stochastic gradient descent. It computes the gradient for small random subsets (mini-batches) of the dataset and updates parameters. It is commonly used when training deep learning models.

Conclusion

Gradient descent is a very important optimisation technique for machine learning models. Its ability to minimise the error function iteratively allows algorithms to improve with each step, resulting in more accurate models. By understanding the differences between the various gradient descent methods, you can adjust your approach to fit each problem, making the training of models faster and more accurate.

Resources:

Machine learning specialisation course on Coursera

Data Preprocessing Techniques for Machine Learning in Python

Ife — Thu, 14 Nov 2024 08:35:14 +0000

Data preprocessing is a critical step in machine learning workflows. It is the act of carrying out certain actions or steps on a dataset to improve the dataset's quality before it is used for machine learning or other tasks. Data preprocessing steps involve cleaning, transforming, normalization and handling outliers in order to improve its quality or ensure that it is suitable for its main purpose (in this case, machine learning). A clean and high-quality dataset enhances the machine learning model's performance.

Common issues with low-quality data include:

Missing values
Inconsistent formats
Duplicate values
Irrelevant features

In summary, these are the steps in data preprocessing for machine learning:

Import necessary libraries.
Load and inspect the dataset.
Data cleaning
- Handling missing values.
- Duplicate removal.
- Dealing with outliers.
Data transformation
- Normalization
- Standardization

You will need basic knowledge of Python and how to use Python libraries for data preprocessing to be able to follow this guide.

Requirements:
The following are required for data preprocessing in this guide:

Python 3.12
Jupyter Notebook or your favourite notebook
Numpy
Pandas
Scipy
Scikit learn
Melbourne Housing Dataset

You can also check out the output of each code in these Jupyter notebooks on GitHub.

Import necessary libraries

If you haven't installed Python already, you can download it from the Python website and follow the instructions to install it.

Once Python has been installed, install the required libraries

pip install numpy scipy pandas scikit-learn

Install Jupyter Notebook.

pip install notebook

After installation, start Jupyter Notebook with the following command

jupyter notebook

This will launch Jupyter Notebook in your default web browser. If not, check the terminal for a link you can manually paste into your browser.

Open a new notebook from the File menu, import the required libraries and run the cell

import numpy as np
import pandas as pd
import scipy

Load and Inspect the Data

Go to the Melbourne Housing Dataset site and download the dataset. Load the dataset into the notebook using the following code. You can copy the file path on your computer to paste in the read_csv function. You can also put the CSV file in the same folder as the notebook and import the file as seen below.

data = pd.read_csv(r"melb_data.csv")

# View the first 5 columns of the dataset
data.head()

Split the data into training and validation sets

from sklearn.model_selection import train_test_split

# Set the target
y = data['Price']

# Firstly drop categorical data types
melb_features = data.drop(['Price'], axis=1) #drop the target column

X = melb_features.select_dtypes(exclude=['object'])

# Divide data into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2, random_state=0)

You have to split data into training and validation sets to prevent data leakage. As a result, whatever preprocessing technique you carry out on the training features set is the same as the one you carry out on the validation features set.

Now the dataset is ready for preprocessing!

Data Cleaning

Handling missing values
Missing values in a dataset are like holes in a fabric that are supposed to be used to sew a dress. It spoils the dress before it is even made.

There are 3 ways to handle missing values in a dataset.

Drop the rows or columns with empty cells

#Drop the rows with empty cells from the original data frame
data.dropna(inplace=True)

#Drop the columns with empty cells
#firstly, get the names of columns with empty cells
cols_with_empty_cells = [col for col in X_train.columns if X_train[col].isnull().any()]

#secondly, drop the columns with the empty cells
removed_X_train_cols = X_train.drop(cols_with_empty_cells, axis=1)
removed_X_valid_cols = X_valid.drop(cols_with_empty_cells, axis=1)

The issue with this method is that you may lose valuable information that you are to train your model with. Unless most values in the dropped rows or columns are missing, there is no need to drop either rows or columns with empty cells.

Impute values in the empty cells

You can impute or fill in the empty cells with the mean, median or mode of the data in that particular column. SimpleImputer from Scikit Learn will be used to impute values in the empty cells

from sklearn.impute import SimpleImputer

# Impute values
imputer = SimpleImputer()
imputed_X_train_values = pd.DataFrame(imputer.fit_transform(X_train))
imputed_X_valid_values = pd.DataFrame(imputer.transform(X_valid))

# Imputation removed column names so we put them back
imputed_X_train_values.columns = X_train.columns
imputed_X_valid_values.columns = X_valid.columns

# Set the imputed values to X_train
X_train = imputed_X_train_values
X_valid = imputed_X_valid_values

X_train.head()

Impute and notify

How this works is that you impute values in the empty cells but you also create a column that indicates that the cell was initially empty.

# Make new columns indicating what will be imputed
# The column will have booleans as values
for col in cols_with_empty_cells:
    X_train[col + '_was_missing'] = X_train[col].isnull()
    X_valid[col + '_was_missing'] = X_valid[col].isnull()

# Impute values
imputer = SimpleImputer()
imputed_X_train_values = pd.DataFrame(imputer.fit_transform(X_train))
imputed_X_valid_values = pd.DataFrame(imputer.transform(X_valid))

# Imputation removed column names so we put them back
imputed_X_train_values.columns = X_train.columns
imputed_X_valid_values.columns = X_valid.columns

# Set the imputed values to X_train
X_train = imputed_X_train_values
X_valid = imputed_X_valid_values

# See the new columns and their values
X_train.head()

Duplicate removal
Duplicate cells mean repeated data and it affects model accuracy. The only way to deal with them is to drop them.

# Check for the number of duplicate rows in the dataset
X_train.duplicated().sum()

# Drop the duplicate rows
X_train.drop_duplicates(inplace=True)
X_valid.drop_duplicates(inplace=True)

Dealing with outliers
Outliers are values that are significantly different from the other values in the dataset. They can be unusually high or low compared to other data values. They can arise due to entry errors or they could genuinely be outliers.

It is important to deal with outliers or else they will lead to inaccurate data analysis or models. One method to detect outliers is by calculating z-scores.

The way it works is that the z-score is used to check if a data point is 3 points or more away from the mean value. This calculation is done for every data point. If the z-score for a data point equals 3 or a higher value, the data point is an outlier.

from scipy import stats
import pandas as pd

# Calculate Z-scores for each row in the training and validation datasets
X_train_zscores = stats.zscore(X_train, axis=1)
X_valid_zscores = stats.zscore(X_valid, axis=1)

# Define the threshold for outlier detection
threshold = 3

# Identify rows in X_train and X_valid with values above the threshold (meaning they're outliers)
outliers_train = np.abs(X_train_zscores) > threshold
outliers_valid = np.abs(X_valid_zscores) > threshold

# Remove rows identified as outliers from X_train and X_valid (~ means NOT)
X_train_no_outliers = X_train[~outliers_train]
X_valid_no_outliers = X_valid[~outliers_valid]

# Display the results
print("Original X_train shape:", X_train.shape)
print("X_train shape after removing outliers:", X_train_no_outliers.shape)

print("Original X_valid shape:", X_valid.shape)
print("X_valid shape after removing outliers:", X_valid_no_outliers.shape)

Data Transformation

Normalization
You normalize features so they can be described as a normal distribution.

A normal distribution (also known as the Gaussian distribution) is a statistical distribution where there are roughly equal distances or distributions above and below the mean. The graph of the data points of a normally distributed data form a bell curve.

The point of normalizing data is if the machine learning algorithm you want to use assumes that the data is normally distributed. An example is the Gaussian Naive Bayes model.

from sklearn.preprocessing import MinMaxScaler

# Initialize the MinMaxScaler
scaler = MinMaxScaler()

# Fit the scaler on the training data and transform it
X_train_normalized = scaler.fit_transform(X_train)

# Transform the validation data using the same scaler
X_valid_normalized = scaler.transform(X_valid)

# Convert the normalized data back into DataFrames to keep column names
X_train_normalized = pd.DataFrame(X_train_normalized, columns=X_train.columns, index=X_train.index)
X_valid_normalized = pd.DataFrame(X_valid_normalized, columns=X_valid.columns, index=X_valid.index)

# Display the results
print("First few rows of normalized X_train:")
print(X_train_normalized.head())

print("First few rows of normalized X_valid:")
print(X_valid_normalized.head())

Standardization
Standardization transforms the features of a dataset to have a mean of 0 and a standard deviation of 1. This process scales each feature so that it has similar ranges across the data. This ensures that each feature contributes equally to model training.

You use standardization when:

The features in your data are on different scales or units.
The machine learning model you want to use is based on distance or gradient-based optimizations (e.g., linear regression, logistic regression, K-means clustering).

You use StandardScaler() from the sklearn library to standardize features.

from sklearn.preprocessing import StandardScaler

# Initialize the StandardScaler
scaler = StandardScaler()

# Fit the scaler on the training and validation data and transform them
X_train_standardized = scaler.fit_transform(X_train)
X_valid_standardized = scaler.transform(X_valid)

# Convert the standardized data back into DataFrames to keep column names
X_train_standardized = pd.DataFrame(X_train_standardized, columns=X_train.columns, index=X_train.index)
X_valid_standardized = pd.DataFrame(X_valid_standardized, columns=X_valid.columns, index=X_valid.index)

# Display the results
print("First few rows of standardized X_train:")
print(X_train_standardized.head())

print("First few rows of standardized X_valid:")
print(X_valid_standardized.head())

Conclusion

Data preprocessing is not just a preliminary stage. It is part of the process of building accurate machine learning models. It can also be tweaked to fit the needs of the dataset you are working with.

Like with most activities, practice makes perfect. As you continue to practise these data preprocessing techniques, your skills will improve as well as your models.

Thank you for reading through. I would love to read your thoughts on this 👇

How to Build a Books CRUD API with FastAPI

Ife — Sat, 26 Aug 2023 00:13:24 +0000

Introduction

FastAPI is a modern Python web framework which has gained popularity for its high-performance capabilities and development time. I like it for its overall simplicity compared to Django.

In this tutorial, I will guide you through building a library CRUD (Create, Read, Update, Delete) application using FastAPI. By the end of this guide, you will have a fully functional books API that allows users to manage books effortlessly.

Prerequisites

Before we dive into the development process, let's ensure you have the necessary tools in place:

Python (3.6+)
FastAPI

You can install FastAPI using the following command:

pip install fastapi

Uvicorn

You can install Uvicorn using the following command:

pip install uvicorn[standard]

Building the CRUD API

Setting Up the Project

Let's start by setting up the project structure:

Create a new directory for your project.
Create a virtual environment and activate it

python -m venv myenv

myenv\Scripts\activate.bat #for Windows
source myenv/bin/activate #for Mac OS and Linux

Install FastAPI and Uvicorn in the virtual environment

pip install fastapi uvicorn[standard]

Inside the project directory, create a file named books.py.

FastAPI provides a lot of features for software development. In books.py, import the required dependencies:

from fastapi import FastAPI, Body

For this tutorial, I will use a book list object with information about the books

BOOKS = [ 
    {'title': 'Jane Eyre', 'author': 'Jane Austen', 'category': 'period drama'},
    {'title': 'Great Expectations', 'author': 'Charles Dickens', 'category': 'period drama'},
    {'title': 'Bourne Idemtity', 'author': 'Robert Ludlum', 'category': 'mystery/thriller'},
    {'title': 'DaVinci Code', 'author': 'Dan Brown', 'category': 'mystery/thriller'},
    {'title': 'The Match Girl', 'author': 'Charles Dickens', 'category': 'tragedy'}
]

Instantiate the FastAPI app. This lets you use all the dependencies that come with FastAPI

app = FastAPI()

Implementing CRUD Operations

CRUD stands for create, read, update and delete and each aligns with the HTTP verbs, post, get, put respectively.

Create (POST)

To create the endpoint for the CRUD operations, you will need app decorator, post attribute and the endpoint you want. In this case, the endpoint /books/create_book is static.

The function that follows takes new_book data of type JSON. Body() tells FastAPI that. It then adds the new data to the book list.

@app.post('/books/create_book')
async def create_book(new_book=Body()):
    BOOKS.append(new_book)

Read (GET)

To get all the values from the BOOKS list, we use get.

@app.get('/books')
async def read_all_books():
    return BOOKS

Update (PUT)

To update an entry in the books list, we use put. The function takes the updated data from the user and loops through the list. If the title of the updated entry matches a title in the list, the matching entry is updated. This means the title can't be changed since we are searching for entries through their titles.

@app.put('/books/update_book')
async def update_book(update_book=Body()):
    for i in range(len(BOOKS)):
        if BOOKS[i].get('title').casefold() == update_book.get('title').casefold():
            BOOKS[i] = update_book

Delete (DELETE)

We use delete to delete the entry that matches the book title the user provides. This endpoint is not static as it has a dynamic parameter {book_title} of type string (str).

@app.delete('/books/delete_book/{book_title}')
async def delete_book(book_title: str):
    for i in range(len(BOOKS)):
        if BOOKS[i].get('title').casefold() == book_title.casefold():
            BOOKS.pop(i)
            break

Running the App

Now that we are done building the app, the next step is to test it.

Uvicorn is the server used to run FastAPI locally on your machine. Use the following command to start the server

uvicorn main:app --reload

Testing the Endpoints

FastAPI comes with automatic docs (Swagger UI and Redoc). You can access the docs through http://127.0.0.1:8000/docs or http://127.0.0.1:8000/redoc. Swagger UI doc looks like this.

You can test the API in the documentation.

The code for this project is live on Github

Conclusion

FastAPI helps developers to build APIs efficiently, and this tutorial has showcased its capabilities. In this tutorial, you learnt how to set up FastAPI routes and test your endpoints. In my next tutorial, I will cover dynamic path parameters and query parameters.

Was this tutorial helpful to you?