Forem: Shamso Osman

Building a Chatbot with Next.js, Cerebras API and Llama 3.1

Shamso Osman — Sun, 10 Nov 2024 14:45:33 +0000

Abstract

An interactive AI-powered Japanese language learning platform built with Next.js and the Cerebras AI API. The project creates a personalized JLPT N5 study experience by combining modern AI conversation capabilities with structured language learning principles. The system provides real-time Japanese text formatting with furigana support and category-based learning modules.

Problem Statement

With my JLPT N5 exam looming just weeks away, I found myself needing a study buddy that could help me practice Japanese at any time. Sure, there are plenty of flashcard apps and textbooks out there, but I wanted something more interactive - something that could give me immediate feedback and adapt to my needs. That's when I decided to combine my programming skills with my Japanese studies, leveraging the new Cerebras AI API to create a personalized AI language tutor.

If you've ever studied Japanese, you know the struggle. Textbooks are static, apps can be rigid, and finding a study partner who's available at 2 AM when you're cramming particles and kanji? Good luck! Plus, most resources either show only romaji (roman letters) or throw you into the deep end with kanji without reading aids.

The Solution

Create an interactive, AI-powered Japanese tutor focused on JLPT N5 preparation.
Implement consistent Japanese text formatting with automatic furigana display.
Provide structured learning paths across vocabulary, grammar, reading, and kanji.
Deliver immediate, contextual feedback to learners.

Technologies Used

To bring this idea to life, I combined a few technologies that allowed me to quickly prototype and build the platform:

Frontend Framework: For the frontend, I used Next.js. Its server-side rendering capabilities and API routes make it easy to build dynamic, responsive apps. Next.js 15.0.3, React 18.2.0
AI Integration: The core of the chatbot is the Llama 3.1-70B model, accessed via Cerebras Cloud SDK.
UI Components:
- FontAwesome icons for visual elements
- Custom CSS modules for styling
State Management: React useState hooks
API Integration: Custom Next.js API routes

System Architecture

Frontend

Single-page application with component-based architecture
Message handling system for chat interface
Quick response component for guided learning
Responsive design with mobile support

Backend

RESTful API endpoint (/api/chat)
Integration with Cerebras AI model
Custom message formatting middleware
Environment-based configuration

Data Flow

User input captured through chat interface
Messages processed through Next.js API route
AI response generated via Cerebras API
Response formatted and displayed with proper Japanese text styling

Implementation

Since I’m preparing for JLPT N5, I focused primarily on kana (Hiragana & Katakana). The chatbot is designed to return answers primarily in kana, with kanji and romaji as needed.

Key Features

Chat Interface

const [messages, setMessages] = useState([]);
const [input, setInput] = useState("");

AI Integration

const completionParams = {
  messages: [
    { role: "system", content: systemPrompt },
    { role: "user", content: userMessage }
  ],
  model: "llama3.1-8b"
};

Japanese Text Formatting

response = response.replace(
  /([\u4E00-\u9FFF]+)\((.*?)\)/g,
  (match, kanji, furigana) => `${kanji}(${furigana})`
);

Challenges Overcome

Implemented a regex-based system for formatting Japanese text with furigana.
Developed dynamic, category-based quick responses to enhance learning engagement.
Designed a fully responsive layout compatible with various screen sizes.
Optimized the AI response handling for improved performance.

Testing and Validation

Conducted manual testing to ensure the chat functionality works as intended.
Tested for cross-browser compatibility to ensure consistent behavior across different browsers.
Validated mobile responsiveness to ensure a seamless experience on smartphones and tablets.
Verified the accuracy of Japanese text formatting and its proper display.
Ensured proper handling of API responses under various conditions.

Results and Evaluation

The system successfully:

Delivers real-time Japanese language instruction for learners.
Maintains consistent and accurate formatting of Japanese text with furigana.
Provides structured learning paths across various language components.
Ensures responsive performance across all devices.

Performance Metrics

Average API response time: ~2-3 seconds
Japanese text formatting accuracy: 100%
Mobile compatibility: Fully responsive design
Browser support: Compatible with all modern browsers

Conclusion

The project successfully creates an interactive learning environment for Japanese language learners, integrating AI capabilities with structured language instruction. Future improvements will include:

Integration of speech synthesis for pronunciation assistance.
Implementation of a progress tracking system for learners.
Expansion of content coverage, particularly in vocabulary and kanji.
Incorporation of a spaced repetition learning system to reinforce knowledge retention.

Code and Documentation

The project repository follows Next.js conventions for structure and organization.
API documentation includes detailed information about integrating with the Cerebras AI model.
Custom CSS modules are used for styling specific components of the UI.
An environment configuration guide is provided for deployment and setup.

Try It Yourself!

Want to check it out or contribute?

GitHub repository

Remember to star the repo if you found this helpful! ⭐

Live Demo - coming soon!

References

Understanding Your Data: The Essentials of Exploratory Data Analysis

Shamso Osman — Sun, 11 Aug 2024 18:17:13 +0000

When working with a new dataset, it's important to explore the data to understand its structure, patterns, and anomalies. This process, known as Exploratory Data Analysis (EDA), helps you get familiar with the data before diving into modeling or drawing conclusions.

Exploratory data analysis is one of the basic and essential steps of a data science project. A data scientist involves almost 70% of his work in doing the EDA of the dataset. Source.

Key aspects of EDA include:

Distribution of Data: Examining the distribution of data points to understand their range, central tendencies (mean, median), and dispersion (variance, standard deviation).
Graphical Representations: Utilizing charts such as histograms, box plots, scatter plots, and bar charts to visualize relationships within the data and distributions of variables.
Outlier Detection: Identifying unusual values that deviate from other data points. Outliers can influence statistical analyses and might indicate data entry errors or unique cases.
Correlation Analysis: Checking the relationships between variables to understand how they might affect each other. This includes computing correlation coefficients and creating correlation matrices.
Handling Missing Values: Detecting and deciding how to address missing data points, whether by imputation or removal, depending on their impact and the amount of missing data.
Summary Statistics: Calculating key statistics that provide insight into data trends and nuances.

Types of Exploratory Data Analysis (EDA)

1. Univariate Analysis

Definition: Focuses on analyzing a single variable at a time.
Purpose: To understand the variable's distribution, central tendency, and spread.
Techniques:

Descriptive statistics (mean, median, mode, variance, standard deviation).
Visualizations (histograms, box plots, bar charts, pie charts).

2. Bivariate Analysis

Definition: Examines the relationship between two variables.
Purpose: To understand how one variable affects or is associated with another.
Techniques:

Scatter plots.
Correlation coefficients (Pearson, Spearman).
Cross-tabulations and contingency tables.
Visualizations (line plots, scatter plots, pair plots).

3. Multivariate Analysis

Definition: Investigates interactions between three or more variables.
Purpose: To understand the complex relationships and interactions in the data.
Techniques:

Multivariate plots (pair plots, parallel coordinates plots).
Dimensionality reduction techniques (PCA, t-SNE). Cluster analysis.
Heatmaps and correlation matrices.

4. Descriptive Statistics

Definition: Summarizes the main features of a data set.
Purpose: To provide a quick overview of the data.
Techniques:

Measures of central tendency (mean, median, mode).
Measures of dispersion (range, variance, standard deviation).
Frequency distributions.

5. Graphical Analysis

Definition: Uses visual tools to explore data.
Purpose: To identify patterns, trends, and data anomalies through visualization.
Techniques:

Charts (bar charts, histograms, pie charts).
Plots (scatter plots, line plots, box plots).
Advanced visualizations (heatmaps, violin plots, pair plots).

How to perform Exploratory Data Analysis ?

Source

In this article I'll demonstrate the process using a sample weather dataset. This will be a hands-on approach, so we'll walk through each step with simple explanations.

Let's get started!

Step 1: Loading the Data

Importing Libraries

# Import Libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

pandas is used for data manipulation and analysis.

numpy provides support for large, multi-dimensional arrays and matrices.

matplotlib is a plotting library for creating static, animated, and
interactive visualizations.

seaborn is built on top of matplotlib and provides a high-level interface for creating attractive and informative statistical graphics

Loading the data

The first step in EDA is loading your data into a DataFrame.
We can do this using pandas.

# Load the dataset
data_frame_name = pd.read_csv("File_path_to_your_csv_file")

the content in the bracket is usually the file path. alternatively you can define file path then call it.

# Load the dataset
File_path = "File_path_to_your_csv_file"
data_frame_name = pd.read_csv(File_path)

You can get a quick overview of the data and its structure by
using the .head() method. It gives us a glance at the first few rows to understand the basic structure — what columns are present, how the data is organized, and any initial impressions you might have about the values. By default, it shows the first 5 rows, including column names.

# Display the first five rows
data_frame_name.head()

# Alternatively you can use a print function. Gives you the same results
print(data_frame_name.head())

but wait, what if you want the last rows instead? We use .tail() method instead.

# Display the last five rows
data_frame_name.tail()

# Alternatively you can use a print function. Gives you the same results
print(data_frame_name.tail())

Hmm, how about both the head and tail of the dataframe? We can do that by simply calling our dataframe.

# Display the first five and last five rows
data_frame_name

# Alternatively you can use a print function. Gives you the same results
print(data_frame_name)

Step 2: Checking the Data's Structure

Understanding the structure means knowing the number of rows, columns, and data types present in the dataset. This can give you clues about the kind of analysis you'll be able to perform.

# Check the structure of the dataset
weather_data.info()

This command will show the total number of entries (rows), the number of columns, and the data type of each column. It also highlights how many non-null values are present in each column.

you can also get the numbers of rows and columns separately using .shape

# Display the number of rows and columns in the DataFrame (rows,cols)
data_frame_name.shape

and the columns using .columns

# Display the column names of the DataFrame
data_frame_name.columns

and the data types using .dtypes

# Display the data types of each column
data_frame_name.dtypes

Step 3: Summarizing the Data

Next, you'll want to get a summary of the numerical columns. This provides an overview of the data's central tendency, dispersion, and shape of the distribution. We do that using .describe(). This command gives you a quick statistical summary of each numeric column, including the mean, standard deviation, minimum, and maximum values. This helps in identifying any outliers or unusual distributions.

# Get a summary of numerical columns
data_frame_name.describe()

Step 4: Identifying Missing Values

Missing values can be tricky—they might represent gaps in data collection, or they might be errors. It's essential to identify and decide how to handle them.

# Check for missing values
data_frame_name.isnull().sum()

Luckily our dataset does not have any missing values

If any columns have missing values, you'll need to decide whether to remove them or fill them with an appropriate value (like the mean or median).

We can also check for any duplicates by

# Check for and count duplicate rows
data_frame_name.duplicated().sum()

Luckily our dataset does not have duplicates

Step 5: Visualizing the Data

Visualization helps you see patterns, trends, and relationships in the data that might not be obvious from raw numbers. How can we represent data this way? Most ways to visualize include:

Histograms: Display the distribution of numerical data.
Scatter plots: Show the relationship between two numerical variables.
Bar charts: Compare categorical data.
Line charts: Visualize data over time.

Let's start with a simple histogram to understand the distribution of a particular column, such as temperature:

# Plot a histogram of the 'Column_name' column
plt.hist(weather_data['Column_name'], bins=30, edgecolor='black') #you can change the color to anything
plt.title('Distribution of Column_name')
plt.xlabel('Column_name')  # Label for the x-axis (horizontal axis)
plt.ylabel('Frequency')  # Label for the y-axis (vertical axis)
plt.show()

bins=30: This divides the data into 30 bins (intervals) for counting frequencies. You can adjust this number.
edgecolor='black': This adds black outlines to the bars for better visual separation.

let's change the edgecolor to white and add gridlines

# Plot a histogram of the 'Column_name' column
plt.hist(weather_data['Column_name'], bins=30, edgecolor='white') #you can change the color to anything
plt.title('Distribution of Column_name')
plt.xlabel('Column_name')  # Label for the x-axis (horizontal axis)
plt.ylabel('Frequency')  # Label for the y-axis (vertical axis)
plt.grid(True) # Add gridlines for better readability
plt.show()

Try using different columns and different visualization methods

Step 6: Finding Correlations

Correlation analysis helps you understand how different variables relate to each other. This is especially useful if you plan to build a predictive model later.

# Compute correlation matrix
correlation_matrix = weather_data.corr()

# Display the correlation matrix
correlation_matrix

But wait, we get an error. Why is that? The .corr() method works only on numerical data. How do we handle this? We extract numerical columns from the dataset by creating a new dataframe. Then try the .corr() method again on the new dataframe that contains the numerical columns only.

# Extract numerical features for correlation
numerical_data = data_frame_name.select_dtypes(include=['number'])

# Compute correlation matrix on numerical data
correlation_matrix = numerical_data.corr()

# Display the correlation matrix
correlation_matrix

This matrix shows how each pair of columns relates. Values close to 1 or -1 indicate a strong relationship, while values near 0 suggest little to no relationship.

A way to visualize it is using heatmaps. Heatmaps are a type of data visualization that use color to represent the values in a matrix or table.

sns.heatmap(data, annot=True, cmap='coolwarm', fmt=".2f")
plt.title("Heatmap Title")
plt.show()

data: This is the 2D dataset (e.g., a correlation matrix) that you want to visualize.
annot=True: This displays the numerical values within each cell of the heatmap.
cmap='coolwarm': This sets the color palette for the heatmap. 'coolwarm' is a common choice, but you can explore other options.
fmt=".2f": This formats the displayed numerical values to two decimal places.

Exploratory Data Analysis Tools

Python: An interpreted, object-oriented programming language with dynamic semantics. Its high-level, built-in data structures, combined with dynamic typing and dynamic binding, make it very attractive for rapid application development, as well as for use as a scripting or glue language to connect existing components together. Python and EDA can be used together to identify missing values in a data set, which is important so you can decide how to handle missing values for machine learning.

R: An open-source programming language and free software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. The R language is widely used among statisticians in data science in developing statistical observations and data analysis.

Remember:

An efficient EDA lays the foundation of a successful machine learning pipeline.
EDA is not just about statistics; it's about understanding the story your data tells.
Visualization is key to uncovering patterns and anomalies.
Domain knowledge is essential for interpreting findings effectively.

By mastering EDA, you lay a strong foundation for building predictive models, making data-driven decisions, and gaining valuable insights from your data.

References

Data Analysis Tools and Techniques

Shamso Osman — Sun, 04 Aug 2024 19:16:49 +0000

I. Introduction

Hi there! I'm a recent graduate who developed a passion for data and its potential to drive decision-making.

In this article I want to explore various tools and techniques used in data analysis.

II. Understanding the Basics

Before diving into techniques and tools, it's crucial to grasp some fundamental concepts:

1. What is data analysis?

Simply put, Data analysis is the practice of working with data to answer questions and draw insights. It involves collecting, processing, and interpreting data to help make informed decisions.

2. The data analysis process

I've found that data analysis typically follows these steps:

a). Define the question:

It involves understanding the problem, identifying the data needed to address it, and defining the metrics or indicators to measure the outcomes.

b). Collect the data:

It involves gathering relevant information from various sources. This can be done through various methods such as surveys, interviews, observations, or extracting from existing databases.

c). Clean the data:

It involves checking the data for errors and inconsistencies and correcting or removing them.

d). Analyze the data:

It involves applying statistical or mathematical techniques to the data to discover patterns, relationships, or trends.

e). Interpret the results:

It involves drawing conclusions and generate insights from your analysis using visual representations such as charts and/or graphs.

f). Communicate findings:

This involves presenting the findings of the analysis in a narrative form that is engaging and easy to understand.

C. Types of data

There are two main types of data:

Quantitative data: Numerical information that can be measured and expressed as numbers (e.g., age, income, temperature).
Qualitative data: Non-numerical information that describes qualities or characteristics (e.g., color, texture, opinions).

III. Essential Data Analysis Techniques

As a beginner, I've focused on learning these fundamental techniques:

1. Descriptive Statistics

This involves summarizing and describing the main features of a dataset. It takes into account past trends and how they might influence future performance. I've learned to:

Measure central tendency: Mean, median, and mode, which represent the typical value.
Measure dispersion: Range, variance, and standard deviation, which describe how spread out the data is.
Analyse Data distribution: Shape of the data (normal, skewed, etc.) using histograms and box plots.

2. Exploratory Data Analysis (EDA)

EDA is about exploring data through visual methods. I've practiced creating various charts and graphs to understand patterns and relationships in data. Most popular used ones include:

Histograms: Display the distribution of numerical data.
Scatter plots: Show the relationship between two numerical variables.
Bar charts: Compare categorical data.
Line charts: Visualize data over time.

3. Inferential Statistics

This technique allows us to make predictions or inferences about a population based on a sample of data. I'm still getting my head around concepts like hypothesis testing which involves determining if a claim about a population is true or false and confidence intervals which involves estimating a range of values that likely contains the true population parameter.

4. Regression Analysis

Regression helps in understanding relationships between variables. A regression model can be linear, multiple, logistic, ridge, non-linear, life data, and more. I've started with simple linear regression, examining how one variable affects another. I'm gradually working my way up to multiple regression, which involves multiple independent variables.

5. Time Series Analysis

This technique is used for analyzing time-stamped data. I'm learning how to identify trends, seasonality, and make forecasts.

IV. Popular Data Analysis Tools

I've experimented with several tools:

1. Spreadsheets

I've used both Microsoft Excel and Google Sheets. They're great for basic data manipulation, simple visualizations, and are often my go-to for quick analyses.

2. Programming Languages

a). Python: This has been my primary focus. I use libraries like pandas for data manipulation, matplotlib and seaborn for visualization, and scikit-learn for machine learning tasks.

b). SQL: I've used SQL for querying databases and extracting specific datasets for analysis.

c). R: While I haven't used R yet, I know it's widely used in statistical computing and graphics. It's on my list to learn in the future.

3. Data Visualization Tools

I'm currently experimenting with Power BI, which allows me to create interactive dashboards and reports.

Other popular tools in this category include:

Tableau:

Known for its user-friendly interface and powerful visualization capabilities.
Qlik:

Offers robust data discovery and analytics features.

4. Statistical Software

While I haven't used these, I'm aware of tools like:

SPSS:

Offers a user-friendly interface for complex statistical analyses.

SAS: SAS is a command driven software package used for carrying out advanced statistical analysis and data visualization offering a wide variety of statistical methods and algorithms customizable options for analysis and output and publication quality graphics.

References

What tools or techniques did I not mention? Feel free to comment.