Forem: Yaswanth Teja

Predicting Stock Prices with AI: A Simple Guide to Using LSTM for Nifty50 Forecasting

Yaswanth Teja — Sat, 31 May 2025 07:18:32 +0000

Introduction

Have you ever wondered if computers can predict stock prices? Well, they can—not perfectly, but they can learn from past data to make educated guesses about future prices. In this blog, we’ll explore how we can use a special kind of artificial intelligence (AI) called Long Short-Term Memory (LSTM) to predict the future prices of the Nifty50, India’s leading stock market index.

Don’t worry if you’re not a tech expert—we’ll explain everything in simple terms, just like teaching a friend!

How Does Stock Prediction Work?

Imagine you’re trying to predict the weather. You’d look at past weather patterns—like temperature, humidity, and rainfall—to guess if it’ll rain tomorrow. Similarly, stock prediction works by analyzing past stock prices to forecast future trends.

Here’s how we do it:

Get Historical Data – We download years of Nifty50 stock prices.
Clean the Data – Remove errors or missing values (like a teacher correcting a messy notebook).
Train the AI Model – Teach the computer to recognize patterns in stock prices.
Make Predictions – Ask the AI to predict future prices based on what it learned.

Now, let’s dive deeper into each step!

Step 1: Getting the Data

We use a library called yfinance (Yahoo Finance) to download Nifty50 stock prices from 2005 to 2025. This gives us a big table with daily prices—like a giant Excel sheet with dates and closing prices.

import yfinance as yf

start_date = '2005-05-16'
end_date = '2025-05-15'
nifty_data = yf.download('^NSEI', start=start_date, end=end_date)

Think of this as downloading a history book of stock prices.

Step 2: Cleaning the Data

Sometimes, data has missing or incorrect entries (like a torn page in a book). We fix this by:

Sorting dates correctly.
Removing or filling missing values.

print(f"Missing values in the dataset: {nifty_data.isnull().sum().sum()}")
nifty_data = nifty_data.sort_index()

This ensures our AI learns from clean, organized data.

Step 3: Scaling the Data (Making Numbers Easier to Work With)

Stock prices can be huge (like ₹20,000), but AI works better with smaller numbers (between 0 and 1). We use MinMaxScaler to shrink the numbers while keeping their relationships intact.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(nifty_close)  # nifty_close = Closing prices

This is like converting kilometers into meters—same distance, just easier to handle.

Step 4: Preparing the Data for AI Learning

AI learns from sequences. Imagine teaching a child to predict the next number in this sequence:

1, 2, 3, …? (Answer: 4)

Similarly, we give the AI sequences of 60 days’ stock prices and ask it to predict the 61st day.

def create_dataset(dataset, time_step=60):
    X, y = [], []
    for i in range(len(dataset) - time_step - 1):
        X.append(dataset[i:(i + time_step), 0])  # 60 days of data
        y.append(dataset[i + time_step, 0])      # Next day's price
    return np.array(X), np.array(y)

X_train, y_train = create_dataset(train_data, time_step=60)

This way, the AI learns patterns like:

If prices rise for 10 days, will they fall soon?
Do big drops usually recover?

Step 5: Building the AI (LSTM Model)

Our AI is an LSTM (Long Short-Term Memory) network — a type of neural network great at learning sequences (like stock prices over time).

We build it like this:

First Layer (50 neurons) – Learns basic patterns.
Second Layer (50 neurons) – Learns deeper trends.
Third Layer (50 neurons) – Makes final predictions.
Dropout Layers – Prevents overfitting (like a student who memorizes answers instead of learning concepts).

model = Sequential()
model.add(LSTM(units=50, return_sequences=True, input_shape=(time_step, 1)))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=1))  # Final prediction

We then train the model using past data, just like a student practicing with old exam papers.

Step 6: Making Predictions

After training, the AI can predict future prices. We test it on unseen data (like a surprise test) to see how well it performs.

train_predict = model.predict(X_train)
test_predict = model.predict(X_test)

We measure accuracy using:

RMSE (Root Mean Square Error) – How far predictions are from real prices.
R² Score – How well the model explains price movements (0% = random guess, 100% = perfect prediction).

train_rmse = math.sqrt(mean_squared_error(y_train_actual, train_predict))
test_r2 = r2_score(y_test_actual, test_predict)

If the AI gets ~90% accuracy, it’s doing well!

Step 7: Predicting Future Prices

Finally, we ask the AI: "What will Nifty50 prices be in the next 30 days?"

We feed it the last 60 days’ data and let it predict day by day.

last_60_days = scaled_data[-60:]
future_predictions = []

for _ in range(30):
    X_future = last_60_days.reshape(1, time_step, 1)
    future_pred = model.predict(X_future)
    last_60_days = np.append(last_60_days[1:], future_pred)
    future_predictions.append(future_pred[0, 0])

We then plot the predictions:

The red line shows what the AI thinks will happen!

Here, the sample size is limited to get better accuracy. We need to train them with larger data.

Conclusion: Can AI Really Predict Stocks?

Yes — but with limitations.

✅ Good at:

Finding patterns in historical data.
Making short-term predictions.

❌ Not perfect at:

Predicting sudden crashes (like COVID-19).
Accounting for unexpected news (elections, wars).

Try It Yourself!

Want to run this code? Copy the full script from above and try it on Google Colab or Jupyter Notebook.

click below link and run the below code

Google Colab


import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Dropout
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import datetime as dt
import math

# Download market data
print("Downloading Nifty50 data...")
start_date = '2005-05-16'
end_date = '2025-05-15'
nifty_data = yf.download('^NSEI', start=start_date, end=end_date)

# Data cleaning and preprocessing
print("\nCleaning and preprocessing data...")
print(f"Missing values in the dataset: {nifty_data.isnull().sum().sum()}")
nifty_data = nifty_data.sort_index()
nifty_close = nifty_data['Close'].values.reshape(-1, 1)

# Scale the data
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(nifty_close)

# Split data into training and testing sets
train_size = int(len(scaled_data) * 0.8)
test_size = len(scaled_data) - train_size
train_data, test_data = scaled_data[0:train_size,:], scaled_data[train_size:len(scaled_data),:]
print(f"\nTraining data size: {train_size}, Testing data size: {test_size}")

def create_dataset(dataset, time_step=60):
    X, y = [], []
    for i in range(len(dataset) - time_step - 1):
        X.append(dataset[i:(i + time_step), 0])
        y.append(dataset[i + time_step, 0])
    return np.array(X), np.array(y)

# Create the dataset with time steps
time_step = 60
X_train, y_train = create_dataset(train_data, time_step)
X_test, y_test = create_dataset(test_data, time_step)

# Reshape input to be [samples, time steps, features] which is required for LSTM
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], 1)
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], 1)

# Build LSTM model
print("\nBuilding LSTM model...")
model = Sequential()
# First LSTM layer with 50 neurons and return sequences=True to stack another LSTM layer
model.add(LSTM(units=50, return_sequences=True, input_shape=(time_step, 1)))
model.add(Dropout(0.2)) # Dropout to prevent overfitting

# Second LSTM layer with 50 neurons
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))

# Third LSTM layer with 50 neurons
model.add(LSTM(units=50))
model.add(Dropout(0.2))

# Output layer
model.add(Dense(units=1))

# Compile the model
model.compile(optimizer='adam', loss='mean_squared_error')

# Early stopping to prevent overfitting
early_stop = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

# Train the model
print("\nTraining the model...")
batch_size = 32
epochs = 50

history = model.fit(
    X_train, y_train,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=0.1,  # Use 10% of training data for validation
    callbacks=[early_stop],
    verbose=1
)

# Make predictions and evaluate the model
print("\nMaking predictions...")
train_predict = model.predict(X_train)
test_predict = model.predict(X_test)

# Inverse transform to get actual values
train_predict = scaler.inverse_transform(train_predict)
test_predict = scaler.inverse_transform(test_predict)
y_train_actual = scaler.inverse_transform(y_train.reshape(-1, 1))
y_test_actual = scaler.inverse_transform(y_test.reshape(-1, 1))

# Calculate performance metrics
train_rmse = math.sqrt(mean_squared_error(y_train_actual, train_predict))
test_rmse = math.sqrt(mean_squared_error(y_test_actual, test_predict))
train_mae = mean_absolute_error(y_train_actual, train_predict)
test_mae = mean_absolute_error(y_test_actual, test_predict)
train_r2 = r2_score(y_train_actual, train_predict)
test_r2 = r2_score(y_test_actual, test_predict)

# Display results
print(f"\nTraining RMSE: {train_rmse:.2f}")
print(f"Testing RMSE: {test_rmse:.2f}")
print(f"Training MAE: {train_mae:.2f}")
print(f"Testing MAE: {test_mae:.2f}")
print(f"Training R^2 Score: {train_r2:.2f}")
print(f"Testing R^2 Score: {test_r2:.2f}")

# Calculate accuracy as a percentage (simplified for this context)
def calculate_accuracy(actual, predicted, threshold=0.01):
    within_threshold = np.abs(actual - predicted) <= threshold * actual
    accuracy = np.mean(within_threshold) * 100
    return accuracy

train_accuracy = calculate_accuracy(y_train_actual, train_predict)
test_accuracy = calculate_accuracy(y_test_actual, test_predict)

print(f"\nTraining Accuracy: {train_accuracy:.2f}%")
print(f"Testing Accuracy: {test_accuracy:.2f}%")

# Visualize the results
print("\nVisualizing results...")
train_dates = nifty_data.index[time_step+1:train_size]
test_dates = nifty_data.index[train_size+time_step:-1]

plt.figure(figsize=(14, 6))
plt.plot(train_dates, y_train_actual, label='Actual Training Data')
plt.plot(train_dates, train_predict, label='Training Predictions')
plt.plot(test_dates, y_test_actual, label='Actual Testing Data')
plt.plot(test_dates, test_predict, label='Testing Predictions')
plt.title('Nifty50 Price Prediction')
plt.xlabel('Date')
plt.ylabel('Price (INR)')
plt.legend()
plt.show()

# Future predictions
print("\nMaking future predictions...")
last_60_days = scaled_data[-60:]
future_predictions = []

for _ in range(30):
    # Reshape data for prediction
    X_future = last_60_days.reshape(1, time_step, 1)
    # Make prediction
    future_pred = model.predict(X_future)
    # Append to the input data
    last_60_days = np.append(last_60_days[1:], future_pred)
    last_60_days = last_60_days.reshape(-1, 1)
    # Store the prediction
    future_predictions.append(future_pred[0, 0])

# Inverse transform to get actual values
future_predictions = np.array(future_predictions).reshape(-1, 1)
future_predictions = scaler.inverse_transform(future_predictions)

# Create future dates
last_date = nifty_data.index[-1]
future_dates = [last_date + dt.timedelta(days=i) for i in range(1, 31)]

# Visualize future predictions
plt.figure(figsize=(14, 6))
plt.plot(nifty_data.index[-100:], nifty_data['Close'].values[-100:], label='Historical Data')
plt.plot(future_dates, future_predictions, label='Future Predictions', color='red')
plt.title('Nifty50 Future Price Prediction')
plt.xlabel('Date')
plt.ylabel('Price (INR)')
plt.legend()
plt.show()

# Print model summary
print("\nModel Summary:")
model.summary()

Final Thoughts

AI is a powerful tool for stock prediction, but it’s not a crystal ball. It helps investors make educated guesses, not certainties.

Would you trust AI for stock advice? Let us know in the comments! 🚀

Happy investing! 📈

Internal Working Of Python

Yaswanth Teja — Fri, 01 Nov 2024 06:26:41 +0000

Here is the full code file: Code

1. Source Code

When you write a Python script, it’s human-readable text. This source code is the starting point for everything.

Your Python source code, written in a .py file, is human-readable. This code defines what your program does, specifying variables, functions, loops, and so on.

2. Compilation to Bytecode (compiler)

When you run a Python program, the first step is to compile the source code to bytecode. This is done by the Python interpreter:

Syntax Check: Ensures there are no syntax errors.
Compilation: Translates the high-level source code into bytecode, a lower-level, platform-independent representation. This bytecode typically resides in .pyc files within the __pycache__ directory.
- Compiler: Python uses an interpreter, but it first compiles your source code into a lower-level form known as bytecode.
Tokenizing: Breaks down your code into small pieces called tokens (like keywords, operators, identifiers).
Parsing: Analyzes the tokens to ensure they follow Python's syntax rules.
Control Flow Graph (CFG): Represents all paths that might be traversed through a program during its execution.
Bytecode Generation: Converts the parsed tokens into bytecode, a set of instructions for the Python Virtual Machine (PVM).

Let’s deep dive into this.

Python Compiler: Even though Python is known as an interpreted language, it does have a compilation step. Here’s the breakdown:

Tokenization:

Breaks down your code into small pieces called tokens (like keywords, operators, identifiers).

Source Code: Starts with your written code.
Tokenizer (Lexer): This breaks the source code into smaller pieces called tokens, like keywords (for, if), operators (+, -), identifiers (variable names), and literals (like numbers or strings).
Parsing: Analyzes the tokens to ensure they follow Python's syntax rules.
Syntax Analysis: The parser takes these tokens and checks them against Python's grammar rules.
Parse Tree: Builds a tree structure from the tokens, representing the grammatical structure of the code.
Semantic Analysis: Ensures the code makes sense in terms of data types, scope, and other context-specific rules.
Control Flow Graph (CFG): Represents all paths that might be traversed through a program during its execution.
- Control Flow Graph: Represents all possible paths that might be taken through the code during execution.
- Nodes and Edges: Each node represents a basic block of code, and edges represent the flow of control from one block to another.
Bytecode Generation: Converts the parsed tokens into bytecode, a set of instructions for the Python Virtual Machine (PVM).
- The bytecode is a more compact, lower-level representation of your source code, optimized for execution. It’s platform-independent, meaning it can be run on any system with a compatible PVM.
- Bytecode: The parsed code is converted into bytecode, a lower-level, platform-independent representation.
- Instruction Set: This bytecode is a set of instructions that the Python Virtual Machine (PVM) can execute. Bytecode is stored in .pyc files in the __pycache__ directory to speed up future executions.

3. Loading Bytecode (Byte code)

After compilation, the Python Virtual Machine loads the bytecode:

Reading from Cache: If the bytecode has been previously compiled and hasn’t changed, it’s read from the cache (__pycache__). This speeds up execution by skipping the compilation step.
- The bytecode is loaded into memory, ready to be executed. The bytecode is then executed by the PVM, interpreting the instructions to perform the program’s tasks.

4. Execution by PVM (PVM)

The PVM now interprets and executes the bytecode:

Instruction Execution: The PVM reads each bytecode instruction and executes it. Each instruction corresponds to a specific operation, like loading a value, performing arithmetic, or calling a function.
Memory Management: Manages allocation and deallocation of memory for variables and objects.

Memory Management in Python:

Reference Counting: Python keeps track of how many references there are to an object in memory. When the reference count drops to zero, the memory occupied by the object can be reclaimed.
Object Allocation: Python objects (like integers, strings, lists) are created in memory when the code is run.
Garbage Collection: Python has a garbage collector that helps manage memory by deallocating memory that is no longer in use (i.e., objects with a reference count of zero).
Memory Pooling: Python uses pools of memory to allocate small objects more efficiently. This pooling helps reduce the overhead of frequently allocating and deallocating small chunks of memory.
Memory Optimization: Python applies various optimizations to minimize memory usage, such as:
- The PVM performs various runtime optimizations to improve efficiency, such as just-in-time (JIT) compilation in some implementations (like PyPy).
- Reusing small integers and interned strings.
- Efficiently managing data structures (e.g., tuples, lists, dictionaries).

Examples:

Bytecode Caching: The PVM caches compiled bytecode to avoid recompiling the source code every time. This speeds up subsequent runs.
Constant Folding: This involves simplifying constant expressions at compile time rather than runtime. For example, 3 * 2 might be precomputed to 6.

So, in summary: the PVM is like an orchestra conductor, seamlessly turning the bytecode into actions that your computer can execute. The beautiful thing about it is that Python code, thanks to the PVM, is portable and can run on different platforms without modification.

How can we see bytecode generated or not?

When you import a Python module, Python compiles the source code into bytecode and stores it in the __pycache__ directory. This helps speed up future imports by avoiding the need to recompile the module each time it's imported.

Here's the process:

First Import: When you first import a module, Python compiles the .py file into bytecode.
pycache Directory: The bytecode is stored in the __pycache__ directory, named something like module_name.cpython-312.pyc. # 312 is the Python version.
Subsequent Imports: On subsequent imports, Python checks the __pycache__ directory for the compiled bytecode and uses it if the source code hasn't changed, speeding up the import process.

Example:

We have byte.py. When we import code from hello_world.py after execution of byte.py, we can see there will be a directory __pycache__ in that particular folder, and we can see .pyc files:

from hello_world import greet

greet("Byte code")

By using py_compile

py_compile module, which allows you to compile Python source files into bytecode files. This is a handy way to speed up script execution for future runs.

In byte.py

import py_compile

py_compile.compile('hello_world.py')

The py_compile module compiles hello_world.py into bytecode.
The resulting bytecode is stored in the pycache directory, creating a file named hello_world.cpython-38.pyc (or similar, depending on your Python version).

Generating Bytecode:

The whole script is executed to generate the bytecode. This means any top-level code (like print("Hello, World!") and print("c")) will run during the compilation process.

Resulting Bytecode:

The bytecode contains all functions, classes, and executable statements, which Python uses to speed up future imports of the script.

dis module

The dis module in Python is used to disassemble bytecode into a more readable form. This can help you understand what your Python code is doing under the hood. It’s especially useful for debugging or learning about the internals of Python.

In internal.py we have

# Import the dis module which supports the analysis of CPython bytecode
import dis

# Define a simple function to illustrate Python's internals
def greet(name):
    # A simple print statement that is more complex than it looks
    print(f'Hello, {name}!')

# Disassemble the greet function to see what's happening under the hood
def disassemble_function():
    dis.dis(greet)

# Call the disassemble function
disassemble_function()

output

 5           0 LOAD_GLOBAL              0 (print)
              2 LOAD_CONST               1 ('Hello, ')
              4 LOAD_FAST                0 (name)
              6 FORMAT_VALUE             0
              8 BUILD_STRING             2
             10 CALL_FUNCTION            1
             12 POP_TOP
             14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

The program begins by importing the dis module, a powerful tool for analyzing CPython bytecode. CPython is the default implementation of Python and bytecode is an intermediate language for the Python interpreter.
Next, I’ve defined a simple function called greet. This function takes in a parameter name and prints out a greeting. Though the function itself is quite straightforward, what happens under the hood in Python is more intricate than it might seem on the surface.
The disassemble_function function makes use of dis.dis() to disassemble the greet function. dis.dis() translates Python functions into the low-level bytecode that Python’s virtual machine actually executes. This bytecode is Python’s interpretation of our greet function and is a step closer to machine code.
When the script calls disassemble_function(), the console output presents the bytecode of our greet function.

Here’s what the bytecode tells us:

LOAD_GLOBAL(0): This opcode is used to load the global variable, which, in this case, is the print function.
LOAD_CONST(1): This loads the constant value ‘Hello, ‘ onto the stack.
LOAD_FAST(0): This opcode loads the local variable name onto the stack.
FORMAT_VALUE(0): This formats our name string, preparing it to be inserted into the string that’s about to be built.
BUILD_STRING(2): This takes the top two values on the stack (‘Hello, ‘ and name) and builds the final string.
CALL_FUNCTION(1): This line calls the function (global print function that we loaded onto the stack), with the argument count in parentheses (we have one argument, our formatted string).
POP_TOP: This removes the top of the stack (the result of the previous call, since print returns None).
LOAD_CONST(0): Loads None.
RETURN_VALUE: This is the return value of the greet function, which, since there is no explicit return statement, is None.
In essence, the bytecode shows the individual operations that Python performs to execute our greet function. Understanding these instructions is crucial for developers to understand how Python executes code, optimizes functions, and manages resources—all of this happening seamlessly under the hood when we run our Python code.

Ain’t that a delightful dive into the Python machine room? Keep coding and keep exploring the depths of this language’s engine room🚀!

Confusion Matrix: A Clear Guide to Understanding It

Yaswanth Teja — Tue, 11 Jun 2024 08:27:04 +0000

Are you struggling to understand the Confusion Matrix? You're not alone. Despite its name, the Confusion Matrix can be quite perplexing for many. However, we're here to simplify it for you.

What is a Confusion Matrix?

A Confusion Matrix is a crucial tool in machine learning and statistics. It helps you evaluate the performance of a classification algorithm. By breaking down the true positives, true negatives, false positives, and false negatives, you can gain a clear picture of how well your model is performing.

Why is the Confusion Matrix Important?

Performance Measurement: It provides detailed insight into the performance of your classification model.
Error Analysis: Helps identify where your model is making mistakes, allowing for targeted improvements.
Model Comparison: Essential for comparing different models to select the best one.

Why is it called a "Confusion" Matrix?

Because it shows where the model gets "confused" in its predictions.

How to Interpret a Confusion Matrix?

True Positive(TP): Positive events are correctly classified as Positive.
True Negative(TN): Negative events are correctly classified as Negative.
False Positive(FP): (Type 1 Error) Negative values are incorrectly classified as Positive.
False Negative(FN): (Type 2 Error) Positive values are incorrectly classified as Negative. we discuss these more clearly as we move further.

The performance of the classification model is evaluated using a confusion matrix. I’m sure most of us have gone through this concept several times and still find it confusing. And realized why it is named confusion matrix.

In supervised learning, if the target variable is categorial, then the classification model has to classify given test data into respective categories. The efficiency of classified data to respective categories is checked using a confusion matrix. For simplicity let’s consider binary classification.

The results of the classification model can be categorized into four types:

True Positive(TP): Correctly predicted positive observations.
- Cases where the model correctly predicts the class or events as positive.

Eg1: You think you’ll love the new pizza place,
and you do!

Eg2: A patient has got heart disease and the model predicts the same.

True Negative(TN): Negative events are correctly classified as Negative.
- Cases where the model correctly predicts the class or events as negative.

Eg: You think you won't enjoy that new movie, and you don’t.

Eg: A patient does not have heart disease and the model predicts that the patient is all right.

False Positive(FP): (Type 1 Error)
Incorrectly predicted positive observations (Type I error).
- Cases where the model incorrectly predicts the positive class (Type I error).

Eg: You think that weirdly flavored ice cream will be great, but it’s a disaster.

Eg: A patient does not have heart disease but the model predicts the patient has heart disease.

False Negative(FN) : (Type 2 Error)

Incorrectly predicted negative observations (Type II error).

Cases where the model incorrectly predicts the negative class (Type II error).

Eg: You skip that boring-looking book, only to find out later it’s amazing.

Eg: A patient has heart disease but the model predicts as the patient does not have heart disease.

Construction of Confusion Matrix

Here are a few steps to write the confusion matrix: Let’s consider two characters XY where X can be True or False and Y can take the value Positive or Negative.

Step 1: Fill in the confusion matrix for character Y. It is solely based on predicted value, if the prediction is true then it’s Positive otherwise it’s Negative.

Step 2: Fill in the first character X. Compare the actual with the predicted if both are of the same category say Actual is True and Predicted is True fill T (i.e. True), else fill F (i.e. False).

Now it’s time to check the performance of classification using count of TP, FN, FP, and TN.

Evaluation metrics

Consider there are nine Positive and nine Negative actual values out of which few are incorrectly classified by the model.

Key Metrics Derived from the Confusion Matrix:

Accuracy: (TP + TN) / (TP + TN + FP + FN)
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
F1 Score: 2 * (Precision * Recall) / (Precision + Recall)

Recall /True Positive Rate (TPR)/Sensitivity: It tells us among actual true events how many are correctly predicted as true.

Recall = TP / (TP + FN)

Precision: Out of all events that are predicted as positive how many are actually positive?

Precision = TP / (TP + FP)

Accuracy: It tells, out of all events how many are predicted correctly.

Accuracy = ( TP + TN) /(TP + TN + FP + FN )

Accuracy gives the same importance to positive and negative events hence use this when we have a balanced data set. Otherwise, the majority might bias the minority class.

The higher the value of Recall, Precision, and Accuracy better the model.

Misclassification rate (Error Rate): How many are wrongly classified w.r.t total number of values.

Error Rate = (FP + FN) / (TP + TN + FP + FN)

False Positive Rate(FPR)/ Specificity: The number of values that are wrongly classified as positive w.r.t the total number of actual negative values.

FPR = FP / (FP + TN)

F Score: There is a chance that precision may be low while recall is high or vice versa, in this scenario, we need to consider both recall and precision to evaluate the model. Hence F Score comes into existence.

F score = ( 1 + β ^2)* (Precision)* (Recall)/((Precision * β ^2) + Recall)

β factor provides the weightage to Recall and Precision.

β = 1: Recall and Precision are balanced
β < 1: Precision oriented evaluation
β >1: Recall-oriented evaluation

When β = 1 we call it F1 score:

F1 score = 2 * Recall * Precision /(Recall + Precision)

The higher the F1 score better the model.

Implementation of the confusion matrix

import numpy
from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score, precision_score,
recall_score, f1_score

actual = [1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0]
predicted = [1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,1,1,0]

confusion_matrix = metrics.confusion_matrix(actual, predicted)

# Finding precision and recall

accuracy = accuracy_score(actual, predicted)
print("Accuracy :", accuracy)

precision = precision_score(actual, predicted)
print("Precision :", precision)

recall = recall_score(actual, predicted)
print("Recall :", recall)

F1= f1_score(actual, predicted)
print("F1 Score:", F1)

import seaborn as sns
import matplotlib.pyplot as plt
sns.heatmap(confusion_matrix,
annot=True,
fmt='g',
xticklabels=['0', '1'],
yticklabels=['0', '1'])
plt.ylabel('Actual',fontsize=13)
plt.xlabel('Predicted',fontsize=13)
plt.title('Confusion Matrix',fontsize=17)
plt.show()

The trade-off between Recall and Precision:

In real-time scenarios, where it’s okay to have a false positive than a false negative we choose Recall. Eg: Diagnosis of cancer, if the patient has cancer but is diagnosed as negative then the patient’s life is at stake.

Where it’s okay to have false negatives but not false positives we use Precision. Eg: Missing to punish a criminal is better than punishing an innocent person.

Conclusion

Understanding the Confusion Matrix is essential for anyone working with classification models. By demystifying its components and applications, we hope to make it less confusing and more useful in your data analysis toolkit.

EndNote:

I hope you enjoyed the article and gained insight into how to construct the confusion matrix and evaluation matrix like Recall, Precision, F score, and Accuracy. Please drop your suggestions or queries in the comment section.

Alternative to ChatGPT: Introducing Nvidia's Chat with RTX: Revolutionizing the AI Chatbot Experience and Free for Everyone.

Yaswanth Teja — Sat, 06 Apr 2024 10:12:54 +0000

In today's tech-savvy world, Artificial intelligence (AI) is redefining human interactions with technology by pushing new frontiers. As a pioneer in AI computing, Nvidia has lifted the bar once more with the release of Chat with RTX, a game-changing program that looks to revolutionize how humans interact with AI chatbots.

Chat with RTX: What is it?

A cutting-edge software program called Nvidia's Chat with RTX allows customers to run their own AI chatbot on their desktop. Chat with RTX functions both locally and offline, providing users with speed and anonymity, in contrast to conventional chatbot platforms that depend on cloud-based servers.

Important characteristics:

Local Function:

One of Chat with RTX's most notable features is its local functionality. There is no need for internet connectivity because the entire chatbot operates directly on your PC. This assures data confidentiality and privacy in addition to lightning-fast performance.

2.YouTube Helper:

Chat with RTX includes a robust YouTube Helper feature that can read transcripts of videos and offer wise responses to questions posed by users. To ask a question, just copy and paste the URL of a YouTube video into the program. Chat with RTX has you covered whether you're looking for information or clarification on a video.

AI Model Selection:

Users are able to select their favorite AI model from a list of options that includes Ilama and Mistral. Using Mistral 7B,You may be confident of excellent conversational capabilities and accuracy because Mistral 7B is a well-liked option among consumers.

Dataset Options:

Chat with RTX provides users with an array of options for datasets, such as predefined AI model datasets, YouTube video URLs, and local folder directories. Because of its adaptability, users can customize their chatbot experience to meet their own requirements and tastes.

Privacy Assurance:

Chat with RTX provides comfort to people that value their privacy and data protection. Sensitive data is always protected thanks to the software's local operation on the user's PC and the lack of requirement for an online connection.

System Requirements

How to Get Started:

It's simple to get started with Chat with RTX:

Go to https://nvidia.com/en-us/chatrtx/ai-on-rtx.
Download and install the software on your Windows PC.
Begin to take advantage of the advantages of having your very own AI chatbot at your disposal.

In summary:

AI chatbot technology has advanced significantly with Nvidia's Chat with RTX. Users are enabled to fully utilize AI in their daily lives by the software's unmatched privacy, speed, and customization possibilities. A world of opportunities for meaningful and significant AI engagement is made possible by Chat with RTX, regardless of your interests as a tech enthusiast, content provider, or inquisitive explorer. Why then wait? With Chat with RTX, you can already experience the AI chatbots of the future.

Unlocking Code Clarity: A Guide to Effective Type Hinting in Python

Yaswanth Teja — Sun, 25 Feb 2024 10:50:42 +0000

Oh, look, the Python dev finally discovered type hinting! Now they can pretend to be as statically typed as Java, but with extra parentheses and indentation to spare!

Ha! While JavaScript devs are still lost in the dark ages of type ambiguity, still trying to figure out their types while we Python devs are over here adding type hints for fun.

I'm joking, of course!

In the ever-evolving landscape of Python programming, one concept that has gained significant traction in recent years is type hinting. type hinting has emerged as a pivotal concept, enhancing code clarity and maintainability.

In this blog post, we'll delve into the world of type hinting, exploring what it is, and why it matters, Join us as we explore the significance of type hinting and its effective application in Python projects.

What is Type Hinting?

In the dynamic world of Python programming.

Type hinting is a feature introduced in Python 3.5 in Python that allows you to add type information to your code. While Python is a dynamically typed language, meaning you don't have to explicitly declare the type of a variable, type hinting provides a way to indicate the expected types for variables, function arguments, and return values.

While type hints are entirely optional and do not affect the runtime behavior of Python code, they offer several benefits in terms of code readability, documentation, and static analysis.

Python is a Dynamically Typed language

Another staple of dynamic languages is that you don't need to declare a variable before using it.

You can just assign whatever you want to whatever variable you want.

Is this a good thing or a bad thing?

The dynamism of Python gives programmers a lot of power!

But, with great power, comes great responsibility!

And many bugs!

But what does that look like?

Let us start by looking at the function multiply from the beginning with and without type hints.

The bottom-right adds type hints:

👉 : int and : str to the parameters; and
👉 → str to the return value.

Type hints are an OPTIONAL indication:

👉 factor: int says “I expect this to be an integer”;
👉 string: str says “I expect this to be a string”; and
👉 → str says “I expect the function to return a string”.

Type hinting can make your code more readable, help catch potential errors early in development, and improve collaboration, especially in larger codebases. One popular tool for static type checking in Python is mypy.

Here's a brief overview of how you can use type hinting and mypy:

Type Hinting Basics:

Variable Type Hinting:

age: int = 25
name: str = "python"

Function Argument and Return Type Hinting:


def greet(name: str) -> str:
    return f"Hello, {name}"

Type Hints for Collections:


from typing import List, Dict

def process_data(numbers: List[int]) -> Dict[str, int]:
    result = {'sum': sum(numbers), 'count': len(numbers)}
    return result

Using `mypy` for Static Type Checking:

Installation:
```
pip install mypy
```
Running mypy:

Save your Python script with type hints (e.g., example.py) and run:

If there are no type errors, mypy will not produce any output.

If there are issues, mypy will provide error messages.

Example:


# example.py

def add_numbers(x: int, y: int) -> int:
    return x + y

result: int = add_numbers(5, "7")  # Type error: Argument 2 to "add_numbers" has incompatible type "str"; expected "int"

In this example, mypy would catch the type error, indicating that the second argument to add_numbers is expected to be an int, but a str was provided.

Using mypy is optional, and Python will still execute the code even if there are type errors. However, incorporating type hinting and static type checking can be a valuable practice for improving code quality and catching potential issues early in development.

Type hints act as high-level documentation.

Looking at a variable and knowing its type is better than knowing nothing about it.

Type hints will also warn you when you misuse a variable or a function, which is incredibly useful in complex programs.

These are just 2 advantages.
If you want to learn about type hinting in Python 🐍, you're in for an interesting ride.

You should also DEFINITELY take a look at:

👉 the module typing from the standard library; and
👉 the tool mypy, will check the types in your code.

Here's a small typed program:

Conclusion

In conclusion, type hinting is a valuable addition to the Python developer's toolkit, empowering them to write clearer, more expressive code while leveraging the full potential of Python's dynamic nature. Whether you're working on a small script or a large-scale application, incorporating type hints can elevate the quality and maintainability of your Python code.

Install VS Code in an Android Phone?

Yaswanth Teja — Sun, 03 Sep 2023 16:48:02 +0000

If you wants to code on your Android device using VS Code, this might be the best setup for you.

Follow the steps below:

Step 1 - Install termux

In order to install VS code, you will have to install the Termux app by scrolling down and downloading apk using this link from f-droid.

The official release available on the play store doesn't seem to get the packages updated for some reason.

Termux from Play Store won't work. This is what the official Termux Github repository has to say:

"Termux and its plugins are no longer updated on Google Play Store due to Android 10 issues and have been deprecated. The last version released for Android >= 7 was v0.101. It is highly recommended to not install Termux apps from Play Store anymore."

Open the downloaded apk by clicking "Open"

Now Install the apk file. If you are prompted to go to settings and enable installation of Apps from unknown sources, do that!

Open the Termux app and you will see a screen like this:

Step 2 - Install Ubuntu using Termux

Enter the following command on Termux to update the package repository

pkg update

When prompted, press 'y' and then press enter. You will have to press y followed by enter every time you are prompted for it.

Now let's upgrade the packages using the below command in Termux

pkg upgrade

Now let's install proot-distro using the following command:

pkg install proot-distro

Now let's list all the distros we can install using proot using the following command (This command is optional)

proot-distro list

You will see a screen like this:

Now fire this command and Ubuntu will start to install on your Android phone

proot-distro install ubuntu

Ubuntu will start installing and you will see a screen like this:

Now, start Ubuntu by firing the following command:

proot-distro login ubuntu

You will now see a root@ubuntu prompt in the terminal like this:

Step 3 - Downloading code server

While on Ubuntu run the following command:

apt update

and following command to upgrade the package repository:

apt upgrade

Press 'y' and enter whenever prompted.

Now install wget using the following command:

apt install wget

Again press 'y' and enter whenever prompted to confirm the installation. You now have wget installed!

We will now download the latest release of the code server from Github using the following command:

wget https://github.com/coder/code-server/releases/download/v4.16.1/code-server-4.16.1-linux-arm64.tar.gz

Extract the tarball using the following command:

tar -xvf ./code-server-4.16.1-linux-arm64.tar.gz

The tarball will start extracting. Wait for the tarball extraction to finish

Enter the /bin folder of your code-server installation on Ubuntu (running on your phone)

cd code-server-4.16.1-linux-arm64
cd bin

Step 4 - Set up a password and start using VS Code

Setup a password for your VS Code instance using the following command:

export PASSWORD="password"

Note: I am using a very weak password for demonstration purposes as I will use this code-server locally on my phone. Consider using strong passwords for your serious projects

Launch the code server using the following command:

./code-server

Now go to your browser. I am using Google Chrome for Android and go to localhost:8080

You will finally see a screen like this:

After entering the password you will see a welcome screen like this:

You can now start coding in VS Code on your Android Devices

Hope that worked for your phone. Please share your experiences, ask questions, or suggest improvements in the comments section. Happy Hacking!

Alternative

You can use vscode directly by visiting vscode.dev in your browser.

Caveats:

If the update command doesn't work on your Termux or you get some other network errors, try to change the default repo by using the command below:

termux-change-repo

Change the repository to Mirror by Grimle and try the pkg update command again. It should work!

Whenever prompted, press 'y' and then press enter. You will have to press y followed by enter every time you are prompted for it.

A Beginner’s Guide To Machine Learning

Yaswanth Teja — Mon, 27 Mar 2023 08:32:22 +0000

*Machine Learning *(ML) is a powerful subset of Artificial Intelligence (AI) that enables machines to learn from data and make predictions or decisions without being explicitly programmed. It has been a game changer in many fields, from finance and healthcare to creative industries and marketing.

If you're new to ML and want to get started, follow these steps:

1.Understand the basics: Before diving into ML, it's important to have a solid understanding of statistics, linear algebra, and programming. You don't need to be an expert, but having a basic understanding of these topics will help you grasp the fundamentals of ML.

2.Choose your language: There are many programming languages used in ML, including Python, R, and Java. Python is the most popular language for ML, thanks to its simplicity and versatility. It has many libraries specifically designed for ML, such as scikit-learn, TensorFlow, and Keras.

3.Choose your platform: There are many platforms available for ML, including Azure Machine Learning, Amazon SageMaker, and Google Cloud ML Engine. These platforms provide pre-built ML models and tools to help you build your own models.

4. Gather your data: ML models require data to learn from. Make sure you have a sufficient amount of clean, structured data to work with. You can use public datasets or collect your own data.

5.Choose your model: There are many types of ML models, including regression, classification, and clustering. Choose the model that best fits your needs.

6.Train your model: Once you have chosen your model, it's time to train it using your data. This involves feeding your data into the model and adjusting the parameters to optimize performance.

7.Test your model: After training your model, test it using a separate set of data to evaluate its accuracy and performance.

8.Deploy your model: Once your model has been trained and tested, deploy it to a production environment where it can be used in real-world applications.

The basic idea behind machine learning is to build algorithms that can receive input data and use statistical analysis to predict an output value within an acceptable range.

At a high level, machine learning algorithms are designed to learn from data. They do this by finding patterns in the data that they can use to make predictions or decisions. For example, a machine learning algorithm might be trained on a dataset of housing prices and be able to predict the sale price of a new house based on factors such as the size of the house, the number of bedrooms, and the location.

There are many different types of machine learning algorithms, each with its own strengths and weaknesses. Some common types of algorithms include:-

Supervised learning algorithms: These algorithms are trained on a labeled dataset, where the correct output is provided for each example in the training data. The algorithm learns from the training data and is able to make predictions on new data.
Unsupervised learning algorithms: These algorithms are trained on an unlabeled dataset, where the correct output is not provided. The algorithm must find patterns in the data on its own and use them to make predictions or cluster the data into groups.
Reinforcement learning algorithms: These algorithms learn by interacting with their environment and receiving feedback in the form of rewards or punishments. The algorithm learns to maximize the reward and avoid punishment in order to make the best decisions.

Machine learning algorithms are commonly used in a wide range of applications, such as image and speech recognition, natural language processing, fraud detection, and recommendation engines.

One of the key challenges in building effective machine learning models is finding the right balance between underfitting and overfitting. Underfitting occurs when the model is too simple and is not able to capture the underlying patterns in the data, leading to poor performance on both the training data and new data. Overfitting occurs when the model is too complex and is able to memorize the training data, but is not able to generalize well to new data.

To avoid overfitting, machine learning models typically use regularization techniques that constrain the model in some way and prevent it from becoming too complex. Common regularization techniques include adding a penalty for large weights in the model, using early stopping to prevent the model from training for too long, and using cross-validation to evaluate the model’s performance on multiple subsets of the training data.

Remember, ML is a vast and complex field, so don't be discouraged if you encounter challenges along the way. With patience and persistence, you can become proficient in ML and unlock its many benefits.

Summary

Overall, machine learning is a powerful tool that allows us to build intelligent systems that can learn from data and make predictions or decisions. By using machine learning algorithms, we can build systems that are able to adapt and improve over time, making them more effective at solving real-world problems.

Python Virtual Environments

Yaswanth Teja — Sun, 05 Feb 2023 14:37:17 +0000

An Introduction

Python is a widely used programming language that is well known for its simplicity, versatility and robustness. One of the key benefits of using Python is that it provides a vast number of libraries and packages for various applications, including web development, scientific computing, data analysis, machine learning, etc. However, managing multiple Python projects and keeping them isolated from each other can be challenging. This is where Python virtual environments come into play.

What are Python Virtual Environments?

A Python virtual environment is an isolated environment for a Python project, which helps in managing dependencies and packages specific to that project. In other words, virtual environments allow you to create a separate environment for each of your Python projects, ensuring that the dependencies and packages of one project do not interfere with the other. This is particularly useful when you are working on multiple projects that have conflicting dependencies and packages.

How to create a Virtual Environment in Python?

Creating a virtual environment in Python is very easy. The first step is to install the virtual environment package, which is called venv. You can install it using the following command:

pip install virtualenv

Once the package is installed, you can create a virtual environment using the following command:

python -m venv myenv

Here, myenv is the name of the virtual environment that you have created. You can replace it with any other name of your choice.

Activating the Virtual Environment

After creating the virtual environment, you need to activate it. This can be done by running the following command:

source myenv/bin/activate

You can replace myenv with the name of the virtual environment that you created.

Deactivating the Virtual Environment

When you are done working with your virtual environment, you can deactivate it by running the following command:

deactivate

Installing Packages in the Virtual Environment

Once the virtual environment is created and activated, you can install any packages that you need for your project using the following command:

pip install package_name

Here, package_name is the name of the package that you want to install.

Exporting the Virtual Environment

Sometimes, you might need to share your virtual environment with others or move it to another system. To do this, you can export the virtual environment using the following command:

pip freeze > requirements.txt

This will create a file named requirements.txt that contains a list of all the packages and their versions that are installed in your virtual environment. You can then share this file with others or move it to another system, and recreate the virtual environment using the following command:

Copy code

pip install -r requirements.txt

Conclusion

In conclusion, Python virtual environments are a crucial tool for Python developers. They help in keeping the dependencies and packages of each project isolated from each other, ensuring that the projects do not interfere with each other. This makes it easy to manage multiple projects and ensures that the projects are running smoothly and efficiently. With this article, we hope that you have a better understanding of Python virtual environments and how they can be used to manage your Python projects.

A complete guide for Markdown files

Yaswanth Teja — Fri, 13 Jan 2023 12:37:08 +0000

What is Markdown?

According to Wikipedia:

Markdown is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber and Aaron Swartz created Markdown in 2004 as a markup language that is appealing to human readers in its source code form. Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.

A markdown file has .md extension and we can write and design text or useful information by following the basic syntaxes. It’s very popular and easy to use.

HTML tags are also supported in markdown.

Use Case

Markdown syntax is used in various popular places nowadays. You can also make websites using markdown files. This can be done using docsify.

Some popular platforms that use Markdown syntaxes and files are as follows -

GitHub
Hashnode
Dev.to
Discord and many other platforms.

Basic Syntaxes

Headings

Syntax:

# H1 - Heading 1
## H2 - Heading 2
### H3 - Heading 3
#### H4 - Heading 4
##### H5 - Heading 5
###### H6 - Heading 6

Output:

H1 - Heading 1

H2 - Heading 2

H3 - Heading 3

H4 - Heading 4

H5 - Heading 5

H6 - Heading 6

Text

Syntax:

Strikethrough Text

Syntax:

Strikethrough uses two tildes. ~~Scratch this.~~

output:
Strikethrough uses two tildes. ~~Scratch this.~~

Blockquote Text

output:

> “I'm unpredictable, I never know where I'm going until I get there, I'm so random, I'm always growing, learning, changing, I'm never the same person twice. But one thing you can be sure of about me; is I will always do exactly what I want to do.”

output:

“I'm unpredictable, I never know where I'm going until I get there, I'm so random, I'm always growing, learning, changing, I'm never the same person twice. But one thing you can be sure of about me; is I will always do exactly what I want to do.”

Bold

Syntax:

**DO NOT UNDERESTIMATE THE POWER OF A PROGRAMMER.**

output:
DO NOT UNDERESTIMATE THE POWER OF A PROGRAMMER.

Italic

Syntax:

*It is Written in Italics*

output:

It is Written in Italics

Bold and Italic

Syntax:

***You Can Combine Bold and Italics***

output:
You Can Combine Bold and Italics

Highlight

To highlight a text, wrap it in == from both sides. For example, the following markdown:
Syntax Example:

I love ==markdown==

output:

I love ==markdown==

Footnote Syntax:

Here's a sentence with a footnote. [^1]

[^1]: This is the footnote.

output:

Emojis

Syntax:

:mango: :lemon: :man: :car:

output:
🥭 🍋 👨 🚗

Links

Embedded Links Into Text

Syntax:

[Github](https://github.com/)

output:
Github

Image

Syntax:

![alt text](imagelink)

output:

Embed YouTube Video

Syntax:

[![Alt Text](Thumbnail Image)](YOUTUBE VIDEO LINK)

output

Embed GIF

Syntax:

![alt-text](gif link)

output

Horizontal Line Syntax:

---

output

Code

This can be written in 2 ways - Inline code and normal code

Syntax for inline code:

Inline `code` has `back-ticks around` it.

Output:

Inline code has back-ticks around it.

Syntax for normal code:

Blocks of Code



```javascript
var s = "JavaScript syntax highlighting";
alert(s);
```




 
```python
s = "Python syntax highlighting"
print s
```




 
```
No language indicated, so no syntax highlighting. 
But let's throw in a tag.
```

HTML Syntax:

<p align="center">
 Yes, you can use allowed raw HTML in mark-down file.
 This is a paragraph aligned in the center.
</p>
```


`Output:`

<p align="center">
 Yes, you can use allowed raw HTML in mark-down file.
 This is a paragraph aligned in the center.
</p>


## Lists

#### Ordered Lists
`Syntax:`


```markdown
1. First item
2. Second item
3. Third item
```


`Output:`

1. First item
2. Second item
3. Third item

#### Unordered Lists
`Syntax:`



```markdown
- First item
- Second item
- Third item
```



`Output:`

- First item
- Second item
- Third item

- To define a sub list
`Syntax:`



```markdown
* data 1
* data 2
    * data 21
    * data 22
* data 3
```


`Output:`
* data 1
* data 2
    * data 21
    * data 22
* data 3

### Task List
`Syntax:`



```
- [x] Write the press release
- [ ] Update the website
- [ ] Contact the media
```


`Output:`

- [x] Write the press release
- [ ] Update the website
- [ ] Contact the media




# Table

`Syntax:`


```markdown
| Fruit | Emoji   |
| ----- | ------- |
| Mango | :mango: |
| Lemon | :lemon: |
```


`Output:`

| Fruit | Emoji   |
| ----- | ------- |
| Mango | :mango: |
| Lemon | :lemon: |



-  ***Table With Alignments***
`Syntax:`



```markdown
| Fruit(left)                 | Emoji(center) |              Taste(right) |
| :-------------------------- | :-----------: | ------------------------: |
| Mango is the king of Fruits |    :mango:    |       Sweet and I love it |
| Lemon is good for health    |    :lemon:    | Sour, mix it in the water |
```


`Output:`

| Fruit(left)                 | Emoji(center) |              Taste(right) |
| :-------------------------- | :-----------: | ------------------------: |
| Mango is the king of Fruits |    :mango:    |       Sweet and I love it |
| Lemon is good for health    |    :lemon:    | Sour, mix it in the water |


# Mathematical Expressions

1.### Inline expressions:
`Syntax:`



```markdown
$<<mathematical expression>>$
```


`Output:`
Replace `<<mathematical expression>>` with your expression.

`Example`

`Syntax:`



```markdown
$\sqrt{3}+1$
```


`Output:`


![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hpwaw31rmz8x6g65ut1z.png)



2. ***Block Expressions***:

`Syntax:`


```markdown
$$<<mathematical expression>>$$
```


`Example`
$$\sqrt{3}+1$$

`output`

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mrj9oxne72ogohqal36t.png)



3. ***Mixed Expressions***:

`Syntax`




```markdown
When $a \ne 0$, there are two solutions to $(ax^2 + bx + c = 0)$ and they are 

$$ x = {-b \pm \sqrt{b^2-4ac} \over 2a} $$
```


`output`

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/whw0c5tter4e996h5a6a.png)




- [Mathematical Equations github](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions)
- [Mathematical formulas](https://en.wikibooks.org/wiki/LaTeX/Mathematics)


### Conclusion
This is a comprehensive list for you all as your guide. Hope all went well and you enjoyed the article.

Hope this helps you. Thank you for reading, and let’s connect!

8 Week SQL Challenge: Case Study #2 Pizza Runner

Yaswanth Teja — Wed, 12 Oct 2022 16:07:57 +0000

Introduction

Danny was scrolling through his Instagram feed when something really caught his eye — “80s Retro Styling 🎸 and Pizza 🍕 Is The Future!”

Danny was sold on the idea, but he knew that pizza alone was not going to help him get seed funding to expand his new Pizza Empire — so he had one more genius idea to combine with it — he was going to Uberize it — and so Pizza Runner was launched!

Danny started by recruiting “runners” to deliver fresh pizza from Pizza Runner Headquarters (otherwise known as Danny’s house) and also maxed out his credit card to pay freelance developers to build a mobile app to accept orders from customers.

Table Relationship

customer_orders — Customers’ pizza orders with 1 row each for individual pizza with topping exclusions and extras, and order time.
runner_orders — Orders assigned to runners documenting the pickup time, distance and duration from Pizza Runner HQ to customer, and cancellation remark. runners — Runner IDs and registration date
pizza_names — Pizza IDs and name
pizza_recipes — Pizza IDs and topping names
pizza_toppings — Topping IDs and name

Case Study Questions

This case study has LOTS of questions — they are broken up by area of focus including:

A. Pizza Metrics
B. Runner and Customer Experience
C. Ingredient Optimisation
D. Pricing and Ratings
E. Bonus DML Challenges (DML = Data Manipulation Language)

Data Cleaning and Transformation

Before I start with the solutions, I investigate the data and found that there are some cleaning and transformation to do, specifically on the

null values and data types in the customer_orders table
null values and data types in the runner_orders table
Alter data type in pizza_names table

Firstly, to clean up exclusions and extras in the customer_orders — we create TEMP TABLE #customer_orders and use CASE WHEN.

SELECT order_id, customer_id, pizza_id, 
  CASE 
    WHEN exclusions IS null OR exclusions LIKE 'null' THEN ' '
    ELSE exclusions
    END AS exclusions,
  CASE 
    WHEN extras IS NULL or extras LIKE 'null' THEN ' '
    ELSE extras 
    END AS extras, 
  order_time
INTO #customer_orders -- create TEMP TABLE
FROM customer_orders;

Then, we clean the runner_orders table with CASE WHEN and TRIM and create TEMP TABLE #runner_orders.

In summary,

pickup_time — Remove nulls and replace with ‘ ‘
distance — Remove ‘km’ and nulls
duration — Remove ‘minutes’ and nulls
cancellation — Remove NULL and null and replace with ‘ ‘

SELECT order_id, runner_id,
  CASE 
    WHEN pickup_time LIKE 'null' THEN ' '
    ELSE pickup_time 
    END AS pickup_time,
  CASE 
    WHEN distance LIKE 'null' THEN ' '
    WHEN distance LIKE '%km' THEN TRIM('km' from distance) 
    ELSE distance END AS distance,
  CASE 
    WHEN duration LIKE 'null' THEN ' ' 
    WHEN duration LIKE '%mins' THEN TRIM('mins' from duration) 
    WHEN duration LIKE '%minute' THEN TRIM('minute' from duration)        
    WHEN duration LIKE '%minutes' THEN TRIM('minutes' from duration)       
    ELSE duration END AS duration,
  CASE 
    WHEN cancellation IS NULL or cancellation LIKE 'null' THEN ''
    ELSE cancellation END AS cancellation
INTO #runner_orders
FROM runner_orders;

Then, we alter the date according to its correct data type.

pickup_time to DATETIME type
distance to FLOAT type
duration to INT type

ALTER TABLE #runner_orders
ALTER COLUMN pickup_time DATETIME,
ALTER COLUMN distance FLOAT, 
ALTER COLUMN duration INT;

Now that the data has been cleaned and transformed, let’s move on solving the questions! 😉

Solution

A. Pizza Metrics

How many pizzas were ordered?

SELECT COUNT(*) AS pizza_order_count
FROM #customer_orders;

Answer:

Total pizzas ordered are 14.
How many unique customer orders were made?

SELECT COUNT(DISTINCT order_id) AS unique_order_count
FROM #customer_orders;

Answer:

There are 10 unique customer orders made.

How many successful orders were delivered by each runner?

SELECT runner_id, COUNT(order_id) AS successful_orders
FROM #runner_orders
WHERE distance != 0
GROUP BY runner_id;

Answer:

Runner 1 has 4 successful delivered orders.
Runner 2 has 3 successful delivered orders.
Runner 3 has 1 successful delivered order.

How many of each type of pizza was delivered?

SELECT p.pizza_name, COUNT(c.pizza_id) AS delivered_pizza_count
FROM #customer_orders AS c
JOIN #runner_orders AS r
 ON c.order_id = r.order_id
JOIN pizza_names AS p
 ON c.pizza_id = p.pizza_id
WHERE r.distance != 0
GROUP BY p.pizza_name;

Answer:

There are 9 delivered Meatlovers pizzas.
There are 3 delivered Vegetarian pizzas.

How many Vegetarian and Meatlovers were ordered by each customer?

SELECT c.customer_id, p.pizza_name, COUNT(p.pizza_name) AS order_count
FROM #customer_orders AS c
JOIN pizza_names AS p
 ON c.pizza_id= p.pizza_id
GROUP BY c.customer_id, p.pizza_name
ORDER BY c.customer_id;

Answer:

Customer 101 ordered 2 Meatlovers pizzas and 1 Vegetarian pizza.
Customer 102 ordered 2 Meatlovers pizzas and 2 Vegetarian pizzas.
Customer 103 ordered 3 Meatlovers pizzas and 1 Vegetarian pizza.
Customer 104 ordered 1 Meatlovers pizza.
Customer 105 ordered 1 Vegetarian pizza.

What was the maximum number of pizzas delivered in a single order?

WITH pizza_count_cte AS
(
 SELECT c.order_id, COUNT(c.pizza_id) AS pizza_per_order
 FROM #customer_orders AS c
 JOIN #runner_orders AS r
  ON c.order_id = r.order_id
 WHERE r.distance != 0
 GROUP BY c.order_id
)

SELECT MAX(pizza_per_order) AS pizza_count
FROM pizza_count_cte;

Answer:

Maximum number of pizza delivered in a single order is 3 pizzas.

For each customer, how many delivered pizzas had at least 1 change and how many had no changes?

SELECT c.customer_id,
 SUM(CASE 
  WHEN c.exclusions <> ' ' OR c.extras <> ' ' THEN 1
  ELSE 0
  END) AS at_least_1_change,
 SUM(CASE 
  WHEN c.exclusions = ' ' AND c.extras = ' ' THEN 1 
  ELSE 0
  END) AS no_change
FROM #customer_orders AS c
JOIN #runner_orders AS r
 ON c.order_id = r.order_id
WHERE r.distance != 0
GROUP BY c.customer_id
ORDER BY c.customer_id;

Answer:

Customer 101 and 102 likes his/her pizzas per the original recipe.
Customer 103, 104 and 105 have their own preference for pizza topping and requested at least 1 change (extra or exclusion topping) on their pizza.

How many pizzas were delivered that had both exclusions and extras?

SELECT  
 SUM(CASE
  WHEN exclusions IS NOT NULL AND extras IS NOT NULL THEN 1
  ELSE 0
  END) AS pizza_count_w_exclusions_extras
FROM #customer_orders AS c
JOIN #runner_orders AS r
 ON c.order_id = r.order_id
WHERE r.distance >= 1 
 AND exclusions <> ' ' 
 AND extras <> ' ';

Answer:

Only 1 pizza delivered that had both extra and exclusion topping. That’s one fussy customer!

What was the total volume of pizzas ordered for each hour of the day?

SELECT DATEPART(HOUR, [order_time]) AS hour_of_day, 
 COUNT(order_id) AS pizza_count
FROM #customer_orders
GROUP BY DATEPART(HOUR, [order_time]);

Answer:

Highest volume of pizza ordered is at 13 (1:00 pm), 18 (6:00 pm) and 21 (9:00 pm).
Lowest volume of pizza ordered is at 11 (11:00 am), 19 (7:00 pm) and 23 (11:00 pm).
What was the volume of orders for each day of the week?

SELECT FORMAT(DATEADD(DAY, 2, order_time),'dddd') AS day_of_week, 
-- add 2 to adjust 1st day of the week as Monday
 COUNT(order_id) AS total_pizzas_ordered
FROM #customer_orders
GROUP BY FORMAT(DATEADD(DAY, 2, order_time),'dddd');

Answer:

There are 5 pizzas ordered on Friday and Monday.
There are 3 pizzas ordered on Saturday.
There is 1 pizza ordered on Sunday.

B. Runner and Customer Experience

How many runners signed up for each 1 week period? (i.e. week starts 2021-01-01)

SELECT DATEPART(WEEK, registration_date) AS registration_week,
 COUNT(runner_id) AS runner_signup
FROM runners
GROUP BY DATEPART(WEEK, registration_date);

Answer:

On Week 1 of Jan 2021, 2 new runners signed up.
On Week 2 and 3 of Jan 2021, 1 new runner signed up.

What was the average time in minutes it took for each runner to arrive at the Pizza Runner HQ to pickup the order?

WITH time_taken_cte AS
(
 SELECT c.order_id, c.order_time, r.pickup_time, 
  DATEDIFF(MINUTE, c.order_time, r.pickup_time) AS pickup_minutes
 FROM #customer_orders AS c
 JOIN #runner_orders AS r
  ON c.order_id = r.order_id
 WHERE r.distance != 0
 GROUP BY c.order_id, c.order_time, r.pickup_time
)

SELECT AVG(pickup_minutes) AS avg_pickup_minutes
FROM time_taken_cte
WHERE pickup_minutes > 1;

Answer:

The average time taken in minutes by runners to arrive at Pizza Runner HQ to pick up the order is 15 minutes.

Is there any relationship between the number of pizzas and how long the order takes to prepare?

WITH prep_time_cte AS
(
 SELECT c.order_id, COUNT(c.order_id) AS pizza_order, 
  c.order_time, r.pickup_time, 
  DATEDIFF(MINUTE, c.order_time, r.pickup_time) AS prep_time_minutes
 FROM #customer_orders AS c
 JOIN #runner_orders AS r
  ON c.order_id = r.order_id
 WHERE r.distance != 0
 GROUP BY c.order_id, c.order_time, r.pickup_time
)

SELECT pizza_order, AVG(prep_time_minutes) AS avg_prep_time_minutes
FROM prep_time_cte
WHERE prep_time_minutes > 1
GROUP BY pizza_order;

Answer:

On average, a single pizza order takes 12 minutes to prepare.
An order with 3 pizzas takes 30 minutes at an average of 10 minutes per pizza.
It takes 16 minutes to prepare an order with 2 pizzas which is 8 minutes per pizza — making 2 pizzas in a single order the ultimate efficiency rate.

What was the average distance travelled for each customer?

SELECT c.customer_id, AVG(r.distance) AS avg_distance
FROM #customer_orders AS c
JOIN #runner_orders AS r
 ON c.order_id = r.order_id
WHERE r.duration != 0
GROUP BY c.customer_id;

Answer:

(Assuming that distance is calculated from Pizza Runner HQ to customer’s place)

Customer 104 stays the nearest to Pizza Runner HQ at average distance of 10km, whereas Customer 105 stays the furthest at 25km.

What was the difference between the longest and shortest delivery times for all orders?

Firstly, let’s see all the durations for the orders.

SELECT order_id, duration
FROM #runner_orders
WHERE duration not like ' ';

Then, we find the difference by deducting the shortest (MIN) from the longest (MAX) delivery times.

SELECT 
    MAX(duration::NUMERIC) - MIN(duration::NUMERIC) AS delivery_time_difference
FROM #runner_orders
WHERE duration not like '% %'

Answer:

The difference between longest (40 minutes) and shortest (10 minutes) delivery time for all orders is 30 minutes.

What was the average speed for each runner for each delivery and do you notice any trend for these values?

SELECT r.runner_id, c.customer_id, c.order_id, 
 COUNT(c.order_id) AS pizza_count, 
 r.distance, (r.duration / 60) AS duration_hr , 
 ROUND((r.distance/r.duration * 60), 2) AS avg_speed
FROM #runner_orders AS r
JOIN #customer_orders AS c
 ON r.order_id = c.order_id
WHERE distance != 0
GROUP BY r.runner_id, c.customer_id, c.order_id, r.distance, r.duration
ORDER BY c.order_id;

Answer:

(Average speed = Distance in km / Duration in hour)

Runner 1’s average speed runs from 37.5km/h to 60km/h.
Runner 2’s average speed runs from 35.1km/h to 93.6km/h. Danny should investigate Runner 2 as the average speed has a 300% fluctuation rate!
Runner 3’s average speed is 40km/h

What is the successful delivery percentage for each runner?

SELECT runner_id, 
 ROUND(100 * SUM
  (CASE WHEN distance = 0 THEN 0
  ELSE 1
  END) / COUNT(*), 0) AS success_perc
FROM #runner_orders
GROUP BY runner_id;

Answer:

Runner 1 has 100% successful delivery.
Runner 2 has 75% successful delivery.
Runner 3 has 50% successful delivery
(It’s not right to attribute successful delivery to runners as order cancellations are out of the runner’s control.)
I will continue with Part A, B and C soon!

C. Ingredient Optimisation
What are the standard ingredients for each pizza?
What was the most commonly added extra?
What was the most common exclusion?
Generate an order item for each record in the customers_orders table in the format of one of the following:
Meat Lovers
Meat Lovers - Exclude Beef
Meat Lovers - Extra Bacon
Meat Lovers - Exclude Cheese, Bacon - Extra Mushroom, Peppers

Generate an alphabetically ordered comma separated ingredient list for each pizza order from the customer_orders table and add a 2x in front of any relevant ingredients

For example: "Meat Lovers: 2xBacon, Beef, ... , Salami"

What is the total quantity of each ingredient used in all delivered pizzas sorted by most frequent first?

D. Pricing and Ratings

If a Meat Lovers pizza costs $12 and Vegetarian costs $10 and there were no charges for changes — how much money has Pizza Runner made so far if there are no delivery fees?
What if there was an additional $1 charge for any pizza extras?

Add cheese is $1 extra

The Pizza Runner team now wants to add an additional ratings system that allows customers to rate their runner, how would you design an additional table for this new dataset — generate a schema for this new table and insert your own data for ratings for each successful customer order between 1 to 5.
Using your newly generated table — can you join all of the information together to form a table which has the following information for successful deliveries?

customer_id
order_id
runner_id -rating -order_time
pickup_time
Time between order and pickup
Delivery duration
Average speed
Total number of pizzas

If a Meat Lovers pizza was $12 and Vegetarian $10 fixed prices with no cost for extras and each runner is paid $0.30 per kilometre travelled — how much money does Pizza Runner have left over after these deliveries?

E. Bonus Questions
If Danny wants to expand his range of pizzas — how would this impact the existing data design? Write an INSERT statement to demonstrate what would happen if a new Supreme pizza with all the toppings was added to the Pizza Runner menu?

8 Week SQL Challenge: Case Study #1 Danny’s Diner

Yaswanth Teja — Tue, 11 Oct 2022 17:19:23 +0000

Introduction

Danny seriously loves Japanese food so in the beginning of 2021, he decides to embark upon a risky venture and opens up a cute little restaurant that sells his 3 favourite foods: sushi, curry and ramen.

Danny’s Diner is in need of your assistance to help the restaurant stay afloat — the restaurant has captured some very basic data from their few months of operation but have no idea how to use
their data to help them run the business.

Problem Statement

Danny wants to use the data to answer a few simple questions about his customers, especially about their

visiting patterns,
how much money they’ve spent, and
which menu items are their favourite. Having this deeper connection with his customers will help him deliver a better and more personalised experience for his loyal customers.

He plans on using these insights to help him decide whether he should expand the existing customer loyalty program — additionally he needs help to generate some basic datasets so his team can easily inspect the data without needing to use SQL.

The data set contains the following 3 tables which you may refer to the relationship diagram below to understand the connection

sales
members
menu

Table Relationship

Case Study Questions

1.What is the total amount each customer spent at the restaurant?
2.How many days has each customer visited the restaurant?
3.What was the first item from the menu purchased by each customer?
4.What is the most purchased item on the menu and how many times was it purchased by all customers?
5.Which item was the most popular for each customer?
6.Which item was purchased first by the customer after they became a member?
7.Which item was purchased just before the customer became a member?
8.What is the total items and amount spent for each member before they became a member?
9.If each $1 spent equates to 10 points and sushi has a 2x points multiplier — how many points would each customer have?
10In the first week after a customer joins the program (including their join date) they earn 2x points on all items, not just sushi — how many points do customer A and B have at the end of January?

Solution

I’m using Microsoft SQL Server and these are the functions used.

Aggregate functions — SUM, MIN, MAX
Numerical functions — TOP
Joins — Inner join, left join
Temp tables (CTE)
Windows function

What is the total amount each customer spent at the restaurant? We use the SUM and GROUP BY functions to find out total spent for each customer and JOIN function because customer_id is from sales table and price is from menu table.

SELECT s.customer_id, SUM(price) AS total_sales
FROM dbo.sales AS s
JOIN dbo.menu AS m
ON s.product_id = m.product_id
GROUP BY customer_id;

Answer:

Customer A spent $76.
Customer B spent $74.
Customer C spent $36.

How many days has each customer visited the restaurant?

Use DISTINCT and wrap with COUNT function to find out number of days customer visited the restaurant.

If we do not use DISTINCT for order_date, the number of days may be repeated. For example, if customer A visited the restaurant twice on ‘2021–01–07’, then number of days may have counted as 2 instead of 1 day.

SELECT customer_id, COUNT(DISTINCT(order_date)) AS visit_count
FROM dbo.sales
GROUP BY customer_id;

Answer:

Customer A visited 4 times.
Customer B visited 6 times.
Customer C visited 2 times.

What was the first item from the menu purchased by each customer?

First, we have to create a CTE using WITH function. In the summary CTE, we use DENSE_RANK and OVER(PARTITION BY ORDER BY) to create a new column rank based on order_date.

I chose to use DENSE_RANK instead of ROW_NUMBER or RANK as the order_date is not time stamped hence, we do not know which item is ordered first if 2 or more items are ordered on the same day.

WITH ordered_sales_cte AS
(
 SELECT customer_id, order_date, product_name,
  DENSE_RANK() OVER(PARTITION BY s.customer_id
  ORDER BY s.order_date) AS rank
 FROM dbo.sales AS s
 JOIN dbo.menu AS m
  ON s.product_id = m.product_id
)

Subsequently, we GROUP BY the columns to show rank = 1 only.

SELECT customer_id, product_name
FROM ordered_sales_cte
WHERE rank = 1
GROUP BY customer_id, product_name;

Answer:

Customer A’s first order are curry and sushi.
Customer B’s first order is curry.
Customer C’s first order is ramen.

What is the most purchased item on the menu and how many times was it purchased by all customers?

SELECT TOP 1 (COUNT(s.product_id)) AS most_purchased, product_name
FROM dbo.sales AS s
JOIN dbo.menu AS m
 ON s.product_id = m.product_id
GROUP BY s.product_id, product_name
ORDER BY most_purchased DESC;SC;

Answer:

Most purchased item on the menu is ramen. Yummy!

Which item was the most popular for each customer?

Again, we create a CTE to rank the number of orders for each product by DESC order for each customer.

WITH fav_item_cte AS
(
 SELECT s.customer_id, m.product_name, 
  COUNT(m.product_id) AS order_count,
  DENSE_RANK() OVER(PARTITION BY s.customer_id
  ORDER BY COUNT(s.customer_id) DESC) AS rank
FROM dbo.menu AS m
JOIN dbo.sales AS s
 ON m.product_id = s.product_id
GROUP BY s.customer_id, m.product_name
)

Then, we generate results where rank of product = 1 only as the most popular product for individual customer.

SELECT customer_id, product_name, order_count
FROM fav_item_cte 
WHERE rank = 1;

Answers

Customer A and C’s favourite item is ramen.
Customer B enjoys all items in the menu. He/she is a true foodie.

Which item was purchased first by the customer after they became a member?

Yeap, you can guess it! We’re creating another CTE.

In this CTE, we filter order_date to be on or after their join_date and then rank the product_id by the order_date.

WITH member_sales_cte AS 
(
 SELECT s.customer_id, m.join_date, s.order_date,   s.product_id,
         DENSE_RANK() OVER(PARTITION BY s.customer_id
  ORDER BY s.order_date) AS rank
     FROM sales AS s
 JOIN members AS m
  ON s.customer_id = m.customer_id
 WHERE s.order_date >= m.join_date
)

Next, we filter the table by rank = 1 to show first item purchased by customer.

SELECT s.customer_id, s.order_date, m2.product_name 
FROM member_sales_cte AS s
JOIN menu AS m2
 ON s.product_id = m2.product_id
WHERE rank = 1;

Answer:

After Customer A became a member, his/her first order is curry, whereas it’s sushi for Customer B.

Which item was purchased just before the customer became a member?

Basically this is a reversed of Question #6. Create a CTE in order

Create new column rank by partitioning customer_id by DESC order_date to find out the order_date just before the customer became member
Filter order_date before join_date.

WITH prior_member_purchased_cte AS 
(
 SELECT s.customer_id, m.join_date, s.order_date, s.product_id,
         DENSE_RANK() OVER(PARTITION BY s.customer_id
         ORDER BY s.order_date DESC) AS rank
 FROM sales AS s
 JOIN members AS m
  ON s.customer_id = m.customer_id
 WHERE s.order_date < m.join_date
)

Then, pull table to show the last item ordered by customer before becoming member.

SELECT s.customer_id, s.order_date, m2.product_name 
FROM prior_member_purchased_cte AS s
JOIN menu AS m2
 ON s.product_id = m2.product_id
WHERE rank = 1;

Answer:

Customer A’s order before he/she became member is sushi and curry and Customer B’s order is sushi. That must have been a real good sushi!

What is the total items and amount spent for each member before they became a member?

First, filter order_date before their join_date. Then, COUNT unique product_id and SUM the prices total spent before becoming member.

SELECT s.customer_id, COUNT(DISTINCT s.product_id) AS unique_menu_item, SUM(mm.price) AS total_sales
FROM sales AS s
JOIN members AS m
 ON s.customer_id = m.customer_id
JOIN menu AS mm
 ON s.product_id = mm.product_id
WHERE s.order_date < m.join_date
GROUP BY s.customer_id;

Answer: Before becoming members,

Customer A spent $ 25 on 2 items.
Customer B spent $40 on 2 items.

If each $1 spent equates to 10 points and sushi has a 2x points multiplier — how many points would each customer have?

Let’s breakdown the question.

Each $1 spent = 10 points.
But, sushi (product_id 1) gets 2x points, meaning each $1 spent = 20 points
So, we use CASE WHEN to create conditional statements
If product_id = 1, then every $1 price multiply by 20 points
All other product_id that is not 1, multiply $1 by 10 points

So, you can see the table below with new column, points.

WITH price_points AS
 (
 SELECT *, 
 CASE
  WHEN product_id = 1 THEN price * 20
  ELSE price * 10
  END AS points
 FROM menu
 )

Using the table above, we SUM the price, match it to the product_id and SUM the total_points.

SELECT s.customer_id, SUM(p.points) AS total_points
FROM price_points_cte AS p
JOIN sales AS s
 ON p.product_id = s.product_id
GROUP BY s.customer_id

Answer:

Total points for Customer A, B and C are 860, 940 and 360.

In the first week after a customer joins the program (including their join date) they earn 2x points on all items, not just sushi — how many points do customer A and B have at the end of January?

Again, we breakdown the question.

Find out customer’s validity date (which is 6 days after join_date and inclusive of join_date) and last day of Jan 2021 (‘2021–01–21’).

WITH dates_cte AS 
(
 SELECT *, 
  DATEADD(DAY, 6, join_date) AS valid_date, 
  EOMONTH('2021-01-31') AS last_date
 FROM members AS m
)

Then, use CASE WHEN to allocate points by dates and product_name.

SELECT d.customer_id, s.order_date, d.join_date, 
 d.valid_date, d.last_date, m.product_name, m.price,
 SUM(CASE
  WHEN m.product_name = 'sushi' THEN 2 * 10 * m.price
  WHEN s.order_date BETWEEN d.join_date AND d.valid_date THEN 2 * 10 * m.price
  ELSE 10 * m.price
  END) AS points
FROM dates_cte AS d
JOIN sales AS s
 ON d.customer_id = s.customer_id
JOIN menu AS m
 ON s.product_id = m.product_id
WHERE s.order_date < d.last_date
GROUP BY d.customer_id, s.order_date, d.join_date, d.valid_date, d.last_date, m.product_name, m.price

Our assumptions are

Day -X to Day 1 (customer becomes member (join_date), each $1 spent is 10 points and for sushi, each $1 spent is 20 points.
Day 1 (join_date) to Day 7 (valid_date), each $1 spent for all items is 20 points.
Day 8 to last day of Jan 2021 (last_date), each $1 spent is 10 points and sushi is 2x points.
Answer:
Customer A has 1,370points.
Customer B has 820 points.

Bonus Questions

Join All The Things
Recreate the table with: customer_id, order_date, product_name, price, member (Y/N)

SELECT s.customer_id, s.order_date, m.product_name, m.price,
CASE
 WHEN mm.join_date > s.order_date THEN 'N'
 WHEN mm.join_date <= s.order_date THEN 'Y'
 ELSE 'N'
 END AS member
FROM sales AS s
LEFT JOIN menu AS m
 ON s.product_id = m.product_id
LEFT JOIN members AS mm
 ON s.customer_id = mm.customer_id;

Rank All The Things

Danny also requires further information about the ranking of customer products, but he purposely does not need the ranking for non-member purchases so he expects null ranking values for the records when customers are not yet part of the loyalty program.

WITH summary_cte AS 
(
 SELECT s.customer_id, s.order_date, m.product_name, m.price,
  CASE
  WHEN mm.join_date > s.order_date THEN 'N'
  WHEN mm.join_date <= s.order_date THEN 'Y'
  ELSE 'N' END AS member
 FROM sales AS s
 LEFT JOIN menu AS m
  ON s.product_id = m.product_id
 LEFT JOIN members AS mm
  ON s.customer_id = mm.customer_id
)
SELECT *, CASE
 WHEN member = 'N' then NULL
 ELSE
  RANK () OVER(PARTITION BY customer_id, member
  ORDER BY order_date) END AS ranking
FROM summary_cte;

Insights

From the analysis, we discover a few interesting insights that would be certainly useful for Danny.

Customer B is the most frequent visitor with 6 visits in Jan 2021.
Danny’s Diner’s most popular item is ramen, followed by curry and sushi.
Customer A and C loves ramen whereas Customer B seems to enjoy sushi, curry and ramen equally. Who knows, I might be Customer B!
Customer A is the 1st member of Danny’s Diner and his first order is curry. Gotta fulfill his curry cravings!
The last item ordered by Customers A and B before they became members are sushi and curry. Does it mean both of these items are the deciding factor? It must be really delicious for them to sign up as members!
Before they became members, both Customers A and B spent $25 and $40.
Throughout Jan 2021, their points for Customer A: 860, Customer B: 940 and Customer C: 360.
Assuming that members can earn 2x a week from the day they became a member with bonus 2x points for sushi, Customer A has 660 points and Customer B has 340 by the end of Jan 2021.

Thank you Danny Ma for the excellent case study! You can find it here and try it yourself.

Explanation of the most important machine learning models

Yaswanth Teja — Sun, 25 Sep 2022 14:32:28 +0000

In this blog, I will briefly explain some of the most commonly asked machine learning models in interviews. I will also list important parameters related to each model and a source to find a detailed explanation of the same topic, so you can dig deeper if and when required.

Machine learning models can be broadly categorized into two categories supervised and unsupervised learning. Further in supervised learning, we have two broad categories regression and classification. The following sections explain each of them briefly to give you the necessary insights.

Note: I am providing models, which I believe are the most common ones and should be prepared before giving any data science interview. However this list is subjective.

Supervised learning

In supervised learning, the data that you use for training the models is “labeled”. This means the output for each input is known. For example, if your model is trying to predict house prices, you might have variables like size of the house, number of floors, etc. When your data is labeled, it means you would also have a variable that contains the house price.

The above example was for regression. Let’s have a close look at regression and classification now.

Classification

In classification, the output of the model is discrete. For example, consider dog vs cat image classification, where we predict whether the image contains the family of dogs or cats. The class (which is the output of the model) will be discrete here i.e. either dog or cat. Now, we will look through the models which are commonly used for classification.

Logistic regression

Don’t get confused; it has the word “regression” in the name, but it is used for classification. Logistic regression uses an equation to create a curve with your data and then uses this curve to predict the outcome of a new observation.

In essence, a logistic equation is created so that the output values can only be between 0 and 1.

Support Vector Machine

Support Vector Machines (SVM) form a boundary between data points for classification. For example, in the case of 2 dimensions, SVM will create a boundary such that the majority of data points of one class fall on one side of the boundary, and most of the other class falls on the other side.

So the goal in SVM is to find the boundary which maximizes the margin (described in the above image).

Important Parameter/Concepts — Kernel, C, Gamma, Margin

Decision Tree

In the decision tree, you basically ask questions about your observation and follow the tree down until you reach an outcome, as shown below.

In the above example, each square is called a node, and more number of nodes here will cause more overfitting of the model on the dataset.

Important Parameter/Concepts — Node, Leaf Node, Entropy, Information Gain

Random Forest

It is an ensemble learning technique that uses multiple decision trees to give the final output. Random forests create multiple decision trees based on bootstrapped datasets of the original dataset and randomly select subsets of the variable at each step of decision trees. During inference, we get outputs from all the decision trees, and finally, we select the output which has maximum votes. Random forests are generally preferred over decision trees as they prevent overfitting.

Important Parameter/Concepts — Number of decision trees, Size of bootstrapped data, Number of random forest feature, and everything else mentioned in decision tree’s section.

Naive Bayes

This model requires a strong foundation in probability. Its working is based on the Bayes theorem.

Basically, we are trying to find out “What is the probability of a particular output (y) given an input (X)”. We assume that each input variable are independent of each other, so P(X|y) can be written as

Further, P(X) (present in the denominator) will remain the same for all possible output (0, 1). So we calculate P(y|X) for all the possible output, and the output with the highest probability will be the final prediction of the model.

Neural Network

One of the essential machine learning models nowadays. It is called a neural network because it is modeled after how the neurons in our brains work. The neural network finds patterns in the dataset, which sometimes even humans can not recognize. They are very powerful and can be of great use, especially when dealing with images, text, and audio.

In the above image, the first two circles represent the input layer, the middle column of 5 circles represents the hidden layer, and the last circle in yellow is the output layer. Each node in the hidden layers represents both a linear function and an activation function.

Important Parameter/Concepts — Learning rate, Loss function, backpropagation, activation function.

Regression

In regression, the output of the model is continuous. For example, consider house price prediction, where we predict the price of a house based on size and number of floors. This house price will be a continuos variable, not discrete.

Linear regression

It is similar to logistic regression. Here we try to find a line that best fits the data. This best-fit line is then used to make predictions about new data points.

The linear regression is nothing but an equation of line or plane. This equation contains a coefficient for each input variable, reflecting each input variable's sensitivity towards the output.

Important Parameter/Concepts — Regression coefficient

Other important models which can be used for regression tasks are decision tree, random forest, neural network, etc. Since I have already covered them above, I will skip them now.

Unsupervised Learning

In unsupervised learning, the data that you use for training models is “Unlabeled”. This means you only have input. In unsupervised learning, we try to find patterns only based on input data. Two main methods used in unsupervised learning include clustering and dimensionality reduction.

Clustering

Clustering involves the grouping or clustering of data points based on some criteria. It is used for customer segmentation, fraud detection, document classification, etc.

Some common clustering techniques include k-means clustering, hierarchical clustering, mean shift clustering, and density-based clustering. While each technique uses different criteria in finding clusters, they all aim to achieve the same thing.

Dimensionality reduction

It involves reducing the number of input variables under consideration by finding/extracting a set of more important variables. There are two ways to reduce the input features by feature elimination or by feature extraction.

One of the most famous methods for dimensionality reduction is Principal Component Analysis (PCA) which is based on feature extraction.

Conclusion

I hope this attempt to summarize all the important machine learning models comes in handy for you all. If you think there is any scope of improvement or you want an explanation about any other model, please let me know. If you would like to keep yourself updated with more topics about machine learning,

Forem: Yaswanth Teja

Predicting Stock Prices with AI: A Simple Guide to Using LSTM for Nifty50 Forecasting

Introduction

How Does Stock Prediction Work?

Step 1: Getting the Data

Step 2: Cleaning the Data

Step 3: Scaling the Data (Making Numbers Easier to Work With)

Step 4: Preparing the Data for AI Learning

Step 5: Building the AI (LSTM Model)

Step 6: Making Predictions

Step 7: Predicting Future Prices

Conclusion: Can AI Really Predict Stocks?

Try It Yourself!

Final Thoughts

Internal Working Of Python

1. Source Code

2. Compilation to Bytecode (compiler)

3. Loading Bytecode (Byte code)

4. Execution by PVM (PVM)

How can we see bytecode generated or not?

Example:

By using py_compile

Generating Bytecode:

Resulting Bytecode:

dis module

output

Confusion Matrix: A Clear Guide to Understanding It

What is a Confusion Matrix?

Why is the Confusion Matrix Important?

Why is it called a "Confusion" Matrix?

How to Interpret a Confusion Matrix?

Construction of Confusion Matrix

Key Metrics Derived from the Confusion Matrix:

Conclusion

Alternative to ChatGPT: Introducing Nvidia's Chat with RTX: Revolutionizing the AI Chatbot Experience and Free for Everyone.

Chat with RTX: What is it?

Important characteristics:

System Requirements

How to Get Started:

Unlocking Code Clarity: A Guide to Effective Type Hinting in Python

What is Type Hinting?

Python is a Dynamically Typed language

Type Hinting Basics:

Using mypy for Static Type Checking:

Example:

Type hints act as high-level documentation.

Conclusion

Install VS Code in an Android Phone?

Step 1 - Install termux

Step 2 - Install Ubuntu using Termux

Step 3 - Downloading code server

Step 4 - Set up a password and start using VS Code

Alternative

A Beginner’s Guide To Machine Learning

Python Virtual Environments

An Introduction

What are Python Virtual Environments?

How to create a Virtual Environment in Python?

Activating the Virtual Environment

Deactivating the Virtual Environment

Installing Packages in the Virtual Environment

Exporting the Virtual Environment

Conclusion

A complete guide for Markdown files

What is Markdown?

Use Case

Basic Syntaxes

Headings

H1 - Heading 1

H2 - Heading 2

H3 - Heading 3

H4 - Heading 4

H5 - Heading 5

H6 - Heading 6

Text

Links

Code

Syntax for inline code:

Syntax for normal code:

Blocks of Code

Using `mypy` for Static Type Checking: