Forem: Ananya S

🚀 I Built an AI-Powered Fest Assistant with Agents, RAG & Planning (Pragyan @ NITT)

Ananya S — Tue, 14 Apr 2026 17:57:01 +0000

I deleted Instagram more than a year ago, and honestly, it saved me from a lot of distractions.

But there was an unexpected downside.

A lot of informal, real-time information — especially during college events — still lives there.

During our college fest, for example:

Event schedules
Last-minute updates
Food stall announcements
Informal activities

…everything gets posted on Instagram.

At the same time:

Fest details and significance are on the official website
Food stall info is on a separate app
The entire 3-day schedule is compressed into a few posts

There’s no single place to get a clear, structured view of everything.

And that’s when it hit me:

Most college fests have websites.
Some even have apps.

But none of them actually help you navigate the fest intelligently.

They give information.
They don’t give guidance.

But I wanted to build something smarter —
an AI assistant that actually understands queries, plans your day, and even helps you find teammates.

So, I built Pragyan Mentor Assistant — an AI-powered system for navigating a techno-managerial fest.

🎯 Problem

During college fests like Pragyan (NIT Trichy):
There are

There are dozens of events, workshops, and shows
Information is scattered across PDFs, sites, and posters
Users don’t know:
- what to attend
- what matches their interests
- how to plan their time
- who to team up with

👉 Traditional apps = static information
👉 I wanted intelligent interaction

💡 Solution

I built a multi-tool AI assistant that can:

🎉 Answer questions about events, workshops, proshows
🍔 Show food stalls & mess menu
🧠 Recommend activities based on user intent
📅 Plan your schedule
🤝 Match you with like-minded participants/Suggest potential teammates (prototype)
📚 Answer fest-related questions using RAG

🧠 System Design

Instead of a simple chatbot, I designed it as a tool-using agent system.

🔹 Tools

fetch_events
fetch_workshops
fetch_food_stall
fetch_mess_menu
pragyan_bot (RAG-based)
smart_recommender
planner
buddy_matcher

🔹 Agent Flow

User query
LLM decides:

Which tool to call
1. Tool executes
2. Response is generated in natural language

📚 Retrieval Approach

This system uses a hybrid retrieval strategy at the system level:

Structured retrieval (keyword-based)
- Direct tool calls for events/workshops
- Fast and deterministic
Semantic retrieval (RAG)
- Vector search over fest documents
- Handles open-ended queries

👉 This combination allows both precision and flexibility

📚 RAG (Retrieval Augmented Generation)

To handle fest knowledge:

Used:
- Text files (events, shows, lectures, FAQs)
Built:
- FAISS vector store
Retrieval:
- Semantic search on query
Response:
- Context-aware answers

🧠 Memory

Using:

InMemorySaver() (LangGraph)

👉 Enables:

remembering user preferences
better recommendations
conversational continuity

🤖 Smart Features

🎯 Recommendations

Understands intent like:

"What should I attend if I like tech and fun?"

📅 Planner Agent

"Plan my next 3 hours"

Generates a structured schedule based on:

time
interests
available events

🤝 Buddy Matching (Prototype)

Matches based on:

interests
level
context (e.g. case study competitions)

Uses a small dataset to demonstrate logic

🖥️ UI

Built with Streamlit:

Chat-based interface
Quick action buttons
Structured responses

🚀 Deployment

Deployed on Render (free tier)
Environment variables for API security

🎥 Demo

👉 https://www.loom.com/share/13f87025a9154a55b80fc240bfc91ba2

🛠️ Tech Stack

Python
LangChain
OpenAI API
FAISS
Streamlit
Render

⚠️ Challenges Faced

RAG retrieval quality (chunking + parsing issues)
Tool selection accuracy
Structuring multi-agent workflow
Deployment + API key handling

🔄 Ongoing Improvements

Some features I’m actively working on:

Adding database-backed user profiles for real buddy matching
Improving RAG with better retrieval and evaluation
Expanding dataset coverage for more complete fest information
Exploring true hybrid retrieval + reranking

📈 What I Learned

Building agents > building chatbots
RAG needs data structuring, not just embeddings
UI matters a lot for perceived intelligence
Deployment and debugging are part of the real challenge

🔗 Links

🔒 Live demo available on request

💭 Final Thoughts

This project made me realize:

👉 The future isn’t just about LLMs
👉 It’s about systems built around them

If you have suggestions or ideas to improve this, I’d love to hear them!

Stop Calling FAISS a Database: The VectorStore vs. VectorDB Showdown🧠⚡

Ananya S — Tue, 17 Mar 2026 16:40:58 +0000

If you’ve been building with LangChain, you’ve probably used Chroma or FAISS and called them "databases." But in a production environment, that distinction could be the difference between a smooth app and a total system crash.

As AI Engineers, we need to know when to use a lightweight VectorStore and when to upgrade to a full Vector Database.

What is a VectorStore? (The Engine)

A VectorStore is a specialized data structure or a local library. Its primary job is simple: Calculate the distance between vectors as fast as possible.

Best for: Prototypes, local research, and small datasets.
Pros: Zero latency (runs in-process), easy to set up, free.
Cons: If your app restarts, your data might vanish (if not saved to disk). It doesn't scale across multiple servers easily.

Popular Choice: FAISS (by Meta). It's incredibly fast but lacks "database" features like user authentication or real-time updates.

from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# 1. Initialize Embeddings
embeddings = OpenAIEmbeddings()

# 2. Create the VectorStore (In-memory)
texts = ["AI is transforming civil engineering", "LangChain is a framework for LLMs"]
vector_store = FAISS.from_texts(texts, embeddings)

# 3. Search (Fast, but only local)
query = "What is LangChain?"
results = vector_store.similarity_search(query)

# 4. Persistence (Manual step required)
vector_store.save_local("my_faiss_index") 
# To use it later, you must load_local() manually

What is a Vector Database? (The Full System)

A Vector Database is a production-ready management system. It uses a vector store under the hood but wraps it in the features we expect from enterprise software.

Best for: Production apps, multi-user systems, and massive datasets (millions of vectors).

The "Extras" you get:

Persistence: Your data lives on a server, not just in your RAM.
Metadata Filtering: The ability to say "Find similar vectors, but only for documents created in 2024."
Scalability: It can handle billions of vectors by spreading them across different "pods" or nodes.

Popular Choice: Pinecone or Weaviate.

from langchain_pinecone import PineconeVectorStore
from pinecone import Pinecone

# 1. Initialize Cloud Client
pc = Pinecone(api_key="YOUR_PINECONE_API_KEY")
index_name = "my-production-index"

# 2. Connect to the Index (Data lives on Pinecone's servers)
vector_db = PineconeVectorStore.from_texts(
    texts=["B.Tech students at NITT are building AI agents"],
    embedding=OpenAIEmbeddings(),
    index_name=index_name
)

# 3. Search (API Call to the cloud)
# Anyone with the API key can now query this from any device
results = vector_db.similarity_search("Who is building agents?")

Key Observation: There is no "save" step. The moment you run from_texts, the data is permanently stored in the cloud. You can delete your local code, and the data remains accessible.

Feature	VectorStore (e.g., FAISS, Chroma)	Vector Database (e.g., Pinecone, Milvus)
Architecture	A library that runs inside your application code.	A standalone distributed system running on a server.
Data Persistence	Mostly In-Memory. Data is lost when the script ends.	Persistent by default. Data is stored on cloud/disk.
Scalability	Limited by your machine's RAM/Disk. Hard to scale.	Built for Horizontal Scaling. Handles billions of vectors.
Multi-tenancy	No built-in support for isolated users.	High. Supports multiple users and isolated indexes.
CRUD Operations	Hard to update specific vectors without rebuilding.	Full Create, Read, Update, Delete support via API.
Metadata	Basic filtering capabilities.	Advanced Metadata Filtering (e.g., Filter by date).
Cost	Free (Uses your local resources).	Tiered. Free tiers available, then paid.

Let's Discuss!
Are you currently using a local store like Chroma or have you made the jump to a cloud database? What's the biggest challenge you've faced with vector scaling? Drop a comment below! 👇

Decoding Embedding Models: Why Your RAG Is Only as Good as Your Vectors 🚀

Ananya S — Mon, 09 Mar 2026 17:36:43 +0000

As an AI Engineer, the first major decision you make in a RAG (Retrieval-Augmented Generation) pipeline isn't which LLM to use—it's which Embedding Model will represent your data. If your vectors are low-quality, your retrieval will fail, and even a top-tier LLM can't save a response based on the wrong context.

🏗️ What exactly is an Embedding?

Embedding models take text tokens and map them into a multi-dimensional coordinate system (vectors).

Dimensions: These represent the "features" the model understands. Different models represent words in vectors of different dimensions.

Semantic Proximity: In a good model, the vector for "King" and "Queen" will be mathematically closer than "King" and "Keyboard."

🌟 Popular Embedding Models: Hugging Face

Hugging Face models are the go-to for privacy, local deployment, and cost-efficiency. Here are the top picks:

1. all-MiniLM-L6-v2

Dimensions: 384
Description: Fast, efficient, and good quality.
Use Case: General purpose; ideal for real-time applications.

2. all-mpnet-base-v2

Dimensions: 768
Description: The best quality in the MiniLM family, though slower than L6. More dimensions lead to an improvement in accuracy.
Use Case: When quality matters more than speed.

3. all-MiniLM-L12-v2

Dimensions: 384
Description: Slightly better than L6 but a bit slower.
Use Case: A solid balance of speed and quality.

4. multi-qa-MiniLM-L6-cos-v1

Dimensions: 384
Description: Optimized specifically for question-answering.
Use Case: Q&A systems and semantic search.

5. paraphrase-multilingual-MiniLM-L12-v2

Dimensions: 384
Description: Supports 50+ languages.
Use Case: Global and multilingual applications.

💰 The OpenAI Standard

If you need massive scale and the highest "reasoning" in your vectors without managing infrastructure:

text-embedding-3-small

Dimensions: 1536
Cost: ~$0.02 per 1M tokens.
Description: Highly cost-effective with improved accuracy over older models. It also supports Matryoshka Representation Learning, allowing you to trim dimensions (e.g., to 512) to save storage costs without losing much performance.

2. text-embedding-3-large

Dimensions: 3072
Cost: ~$0.13 per 1M tokens.
Description: The most powerful model available. It captures incredibly fine-grained nuances in text.
Feature: Like the "small" version, it supports Matryoshka Representation Learning, which means you can shorten the vector to 256 or 1024 dimensions to save database space while keeping most of the accuracy.
Use Case: Enterprise-level research, legal document analysis, or complex medical data.

3. text-embedding-ada-002

Dimensions: 1536
Cost: ~$0.10 per 1M tokens.
Description: The previous industry standard. While still reliable, it is now considered legacy compared to the "v3" family.
Use Case: Mostly seen in older "legacy" AI systems. For any new project in 2026, you should skip this and go straight to text-embedding-3-small.

⚖️ How to Choose?

Strict Privacy/On-Prem? ➔ Hugging Face (Local).

Real-time/Low Latency? ➔ all-MiniLM-L6-v2.

Multilingual Data? ➔ paraphrase-multilingual-MiniLM-L12-v2.

Enterprise Scale & Accuracy? ➔ text-embedding-3-small.

🚀 Conclusion

In 2026, picking the right embedding model is about balancing latency, cost, and accuracy. Don't just pick the one with the most dimensions—pick the one that fits your specific data and hardware.

What's your go-to embedding model for production? Let's discuss in the comments!

🧠 Your LLM Isn’t an Agent — Until It Has Tools, Memory, and Structure (LangChain Deep Dive)

Ananya S — Mon, 02 Mar 2026 17:09:48 +0000

Most “AI apps” today are just:

Prompt → LLM → Text Response

That’s not an agent.

That’s autocomplete with branding.

A real AI agent can:

🛠 Use tools
🧠 Remember context
📦 Return structured outputs
🔁 Reason across multiple steps

With modern LangChain, building this is surprisingly clean.

Let’s build one properly.

🚀 The Architecture of a Real Agent

A production-ready AI agent has four core components:

Model – the brain
Tools – capabilities
Structured outputs – reliability and formatting
Memory – continuity

If you’re missing one of these, you’re not building a system — you’re running a demo.

1️⃣ The Brain: Modern Agent Setup

We start with create_agent() — the current way to build agents in LangChain.

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

Low temperature = more deterministic reasoning.

Now let’s give it capabilities.

2️⃣ Tools: Giving the Agent Superpowers

Tools are just Python functions with clear docstrings.
The docstring matters — it’s how the model decides when to use the tool.

from langchain.tools import tool

@tool
def calculate_revenue(price: float, quantity: int) -> float:
    """Calculate total revenue given price per unit and quantity sold."""
    return price * quantity


@tool
def get_exchange_rate(currency: str) -> float:
    """Get the USD exchange rate for a given currency code."""
    rates = {"EUR": 1.1, "GBP": 1.25}
    return rates.get(currency.upper(), 1.0)

Now we assemble the agent:

agent = create_agent(
    model=llm,
    tools=[calculate_revenue, get_exchange_rate],
    system_prompt="You are a financial analysis assistant."
)

That’s it.

The agent now:

Decides when math is needed
Calls tools autonomously
Observes results
Produces a final answer

No manual routing logic.

3️⃣ Structured Outputs: Stop Parsing Strings

If you're still doing regex on LLM responses, stop.

Modern agents can return structured data using schemas.

from pydantic import BaseModel

class FinancialReport(BaseModel):
    revenue: float
    currency: str
    usd_value: float

Now we enforce structure:

structured_agent = create_agent(
    model=llm,
    tools=[calculate_revenue, get_exchange_rate],
    response_format=FinancialReport,
)

Now when you invoke:

response = structured_agent.invoke({
    "messages": [
        {"role": "user", "content": "I sold 120 units at 50 EUR each. Convert to USD."}
    ]
})

print(response)

You don’t get text.

You get validated data.
Using Pydantic structured output parsers ensures the datatype of the fields is based on how we defined it.

4️⃣ Memory: Making the Agent Stateful

Without memory, every request is a new message.

With memory, your agent becomes a collaborator.

In LangChain, memory can be plugged in via message history.

Example pattern:

chat_history = []

response = agent.invoke({
    "messages": chat_history + [
        {"role": "user", "content": "My product costs 20 USD."}
    ]
})

chat_history.extend(response["messages"])

response = agent.invoke({
    "messages": chat_history + [
        {"role": "user", "content": "Now calculate revenue for 300 units."}
    ]
})

Now the agent remembers:

Product price
Prior discussion
Contextual decisions

Memory transforms isolated responses into evolving workflows.

🧠 What’s Actually Happening Internally?

When you call:

agent.invoke(...)

The agent:

Reads conversation + system prompt
Plans next action
Chooses a tool (if needed)
Executes tool
Feeds result back into reasoning
Produces structured final output

This loop is grounded in tool-calling rather than fragile prompt tricks.

⚠️ Common Mistakes

Things beginner devs get wrong:

❌ Adding too many tools
❌ Writing vague tool descriptions
❌ Not enforcing structured outputs
❌ Forgetting observability/logging
❌ Letting the agent free-run without constraints

Agents are probabilistic planners — not deterministic scripts.

Design them intentionally.

🏗 The Big Shift in How We Build Software

Before agents:

APIs returned static responses
Business logic was deterministic
LLMs were “smart text generators”

After agents:

LLMs orchestrate execution
Tools become capabilities
Structure guarantees reliability
Memory enables continuity

You're no longer building chat interfaces.

You're building goal-driven systems.

🎯 Final Take

If your AI system:

Doesn’t use tools
Doesn’t enforce structure
Doesn’t maintain memory

It’s not an agent.

It’s autocomplete with better marketing.

With modern LangChain, the barrier to real agents is gone.

The question isn’t “Can we build agents?”

It’s:

What workflows are we ready to automate?

Do comment how you build agents and regarding any interesting types of agents you've built!

Understanding LangChain and Vector Embeddings: The Power Duo of Modern AI Applications

Ananya S — Mon, 23 Feb 2026 17:23:19 +0000

Introduction

In the rapidly evolving landscape of artificial intelligence and natural language processing, two technologies have emerged as fundamental building blocks for creating intelligent applications: LangChain and vector embeddings. Together, they form a powerful combination that enables developers to build sophisticated AI systems capable of understanding, reasoning, and generating human-like responses. This post explores both concepts and demonstrates how they work together to create the next generation of AI applications.

What is LangChain?

LangChain is an open-source framework designed to simplify the development of applications powered by large language models (LLMs). It provides a comprehensive set of tools, components, and abstractions that help developers:

Chain together multiple LLM calls and other components
Integrate with external data sources and APIs
Implement memory to maintain context across interactions
Create agents that can make decisions and take actions
Handle complex workflows with ease

At its core, LangChain acts as a bridge between raw LLM capabilities and real-world applications, providing structure and patterns for building production-ready AI systems.

Understanding Vector Embeddings

Vector embeddings are numerical representations of data (typically text, but also images, audio, etc.) in a high-dimensional space. These representations capture semantic meaning and relationships between items:

Semantic similarity: Items with similar meanings have similar vector representations
Dimensionality: Typically 1536, 768, or other dimensions depending on the model
Distance metrics: Cosine similarity or Euclidean distance can measure relatedness

For example, the words "king" and "queen" would have vector embeddings that are close to each other in vector space, while "king" and "banana" would be farther apart.

Common Embedding Models:

OpenAI's text-embedding-ada-002
Sentence-BERT (SBERT)
Hugging Face's all-MiniLM-L6-v2

How LangChain and Vector Embeddings Work Together

The true power emerges when LangChain integrates vector embeddings into its architecture. Here's how they complement each other:

1. Retrieval-Augmented Generation (RAG)

This is perhaps the most impactful combination. LangChain uses vector embeddings to:

Convert documents into vector representations
Store these vectors in vector databases (like Pinecone, Chroma, or FAISS)
Retrieve relevant context when a user asks a question
Pass the retrieved context to an LLM for generating accurate, grounded responses

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# Create embeddings
embeddings = OpenAIEmbeddings()

# Store documents in vector database
vectorstore = Chroma.from_documents(documents, embeddings)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(),
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

# Get answer with context
result = qa_chain.invoke("What is vector similarity search?")

2. Semantic Search and Filtering

Vector embeddings enable LangChain to perform semantic searches rather than just keyword matching:

Find documents that are conceptually similar to a query
Filter results based on semantic relevance scores
Group similar content automatically
Identify duplicate or near-duplicate content

3. Memory and Context Management

LangChain uses embeddings to:

Store conversation history as vectors
Retrieve relevant past interactions based on current context
Maintain long-term memory across sessions
Recognize when similar situations have occurred before

Practical Use Cases

Enterprise Knowledge Base

A company can use LangChain with vector embeddings to create an intelligent knowledge base:

Ingest internal documents, manuals, and policies
Convert all content to vector embeddings
Allow employees to ask natural language questions
Retrieve relevant information and generate comprehensive answers

Customer Support Chatbot

Build a chatbot that:

Understands customer queries semantically
Retrieves relevant support articles and FAQs
Provides accurate, context-aware responses
Learns from past interactions to improve over time

Research Assistant

Create a tool that:

Analyzes academic papers and research documents
Finds connections between different research areas
Summarizes complex topics based on relevant sources
Recommends related papers based on semantic similarity

Implementation Considerations

Choosing the Right Embedding Model

Consider these factors:

Accuracy vs. Cost: OpenAI embeddings are highly accurate but costly; open-source models like all-MiniLM-L6-v2 are free but less accurate
Dimensionality: Higher dimensions capture more nuance but require more storage and computation
Language support: Some models work better for specific languages or domains

Vector Database Selection

Popular options include:

Chroma: Lightweight, easy to set up, great for development
Pinecone: Fully managed, scalable, production-ready
FAISS: High performance, optimized for similarity search

Performance Optimization

Chunking strategy: How you split documents affects retrieval quality
Indexing techniques: HNSW, IVF, or other indexing methods impact speed and accuracy
Hybrid search: Combine vector search with keyword filtering for better results
Caching: Store frequent queries and results to reduce latency

Code Example: Building a Simple RAG System

Here's a complete example showing how to build a basic RAG system with LangChain and vector embeddings:

import os
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# Set up environment
os.environ["OPENAI_API_KEY"] = "your-api-key"

# Load and process documents
loader = TextLoader("knowledge_base.txt")
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings and vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(texts, embeddings)

# Create QA system
llm = ChatOpenAI(temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

# Query the system
query = "Explain vector embeddings in simple terms"
result = qa_chain.invoke(query)
print(f"Answer: {result['result']}")
print(f"Sources: {[doc.metadata for doc in result['source_documents']]}")

Best Practices

Data Preparation

Clean and preprocess text before embedding
Remove noise, standardize formatting
Consider domain-specific preprocessing

Evaluation Metrics

Recall@k: Percentage of relevant documents in top k results
Mean Reciprocal Rank (MRR): Quality of ranking
Precision: Relevance of retrieved results
End-to-end accuracy: How well the final answer addresses the query

Security and Privacy

Be mindful of sensitive data in vector databases
Implement proper access controls
Consider data retention policies
Be aware of embedding model biases

Future Directions

The intersection of LangChain and vector embeddings is rapidly evolving:

Multimodal embeddings: Combining text, images, and audio embeddings
Real-time indexing: Near-instantaneous updates to knowledge bases
Cross-lingual capabilities: Seamless understanding across languages
Personalized embeddings: Tailored to individual users or organizations
Edge deployment: Running embedding models on devices

Conclusion

LangChain and vector embeddings represent a paradigm shift in how we build AI applications. By combining the power of large language models with semantic understanding through vector representations, developers can create systems that truly understand context, retrieve relevant information, and generate meaningful responses.

The beauty of this combination lies in its accessibility – with the right tools and understanding, developers can build sophisticated AI applications without needing deep expertise in machine learning. As these technologies continue to evolve, we can expect even more powerful and intuitive applications that bridge the gap between human intention and machine capability.

Whether you're building an enterprise knowledge base, a customer support system, or a research assistant, the LangChain + vector embeddings combination provides a robust foundation for creating intelligent, context-aware applications that deliver real value to users.

Getting Started Resources

LangChain Documentation: https://python.langchain.com
Hugging Face Embeddings: https://huggingface.co/models?library=sentence-transformers
Pinecone: https://www.pinecone.io

What use cases are you exploring with LangChain and vector embeddings? Share your experiences in the comments below!

🚀 Stop Hallucinating! Build a RAG Chatbot in 5 Minutes with LangChain

Ananya S — Mon, 09 Feb 2026 18:03:48 +0000

Ever asked an AI about something that happened yesterday, only for it to confidently lie to your face? That’s because LLMs are frozen in time—limited by their training data.

Enter RAG (Retrieval-Augmented Generation). It’s like giving your AI an open-book exam. Instead of guessing, it looks up the answer in your documents first.

In this post, we’re building a simple RAG pipeline using LangChain. Let’s dive in! 🏊‍♂️

🔥 The "Big Idea"

RAG works in three simple steps:

Index: Chop your documents into small "chunks" and turn them into math (vectors).

Retrieve: When a user asks a question, find the chunks that match best.

Augment: Stuff those chunks into the prompt and let the AI summarize them.

🛠️The Setup
You'll need a few libraries. Open your terminal and run:

pip install langchain langchain-openai langchain-community chromadb pypdf

💻 The Code
Here is a complete, minimal script to chat with a PDF. Replace "your_api_key" with your actual OpenAI key.

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

# 1. Set your API Key
os.environ["OPENAI_API_KEY"] = "sk-..."

# 2. Load your data (Change this to your PDF path!)
loader = PyPDFLoader("my_awesome_doc.pdf")
data = loader.load()

# 3. Chop it up! (Chunking)
# We split text so the AI doesn't get overwhelmed.
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = text_splitter.split_documents(data)

# 4. Create the "Brain" (Vector Store)
# This turns text into vectors and stores them locally.
vectorstore = Chroma.from_documents(
    documents=chunks, 
    embedding=OpenAIEmbeddings()
)

# 5. Build the RAG Chain
llm = ChatOpenAI(model_name="gpt-4o", temperature=0)
rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff", # "Stuff" all chunks into the prompt
    retriever=vectorstore.as_retriever()
)

# 6. Ask away!
question = "What is the main conclusion of this document?"
response = rag_chain.invoke(question)

print(f"🤖 AI: {response['result']}")

🤔 Why did we do that?
RecursiveCharacterTextSplitter: Why not just feed the whole PDF? Because LLMs have a "context window" (limit). Chunking keeps the info bite-sized and relevant.

ChromaDB: This is our temporary database. It stores the "meaning" of our text so we can search it numerically.

chain_type="stuff": This is the funniest name in LangChain. It literally means "stuff all the retrieved documents into the prompt."

🌟 Pro-Tips for the Road
Overlap matters: Notice chunk_overlap=100? This ensures that if a sentence is cut in half, the context lives in both chunks.

Local Models: Don't want to pay for OpenAI? Swap ChatOpenAI for Ollama and run it 100% locally!

Garbage In, Garbage Out: If your PDF is a messy scan, your RAG will be messy too. Clean your data!

🎁 Wrapping Up

You just built a production-grade logic loop. RAG is the backbone of almost every AI startup today. Whether it's a legal bot, a medical assistant, or a "Chat with your Resume" tool—you now have the blueprint.

What are you planning to build with RAG? Let me know in the comments! 👇

From Words to Meaning: Core NLP Concepts Every Beginner Must Know

Ananya S — Tue, 27 Jan 2026 17:48:10 +0000

In the previous post, we covered the basics of NLP such as tokenization, stemming, lemmatization, and stop words.

In this continuation, we will understand how machines extract meaning from text and represent language numerically.

1. Named Entity Recognition (NER)

Named Entity Recognition (NER) is an NLP technique used to identify and classify real-world entities in text.

Common entity types:

Person
Organization
Location
Date
Time
Money
Percentage

Example sentence:

Elon Musk is the CEO of Tesla and lives in the USA.

NER output:

Elon Musk → PERSON

Tesla → ORGANIZATION

USA → LOCATION

Why NER is important:

Helps extract structured information from unstructured text
Used in resume parsing and document processing
Widely applied in medical and legal NLP systems
Improves search engines and chatbots

2. Bag of Words (BoW)

Bag of Words is one of the simplest techniques to convert text into numbers.

Core idea:

Word order is ignored
Grammar is ignored
Only word frequency matters

Example:

Sentence 1: I love NLP  
Sentence 2: I love AI

Vocabulary:

[I, love, NLP, AI]

Vector representation:

Sentence 1 → [1, 1, 1, 0]

Sentence 2 → [1, 1, 0, 1]

Advantages:

Very easy to implement
Works well for small datasets
Useful as a baseline model

Limitations:

No understanding of context
No semantic meaning
Treats all words as equally important

3. TF-IDF (Term Frequency – Inverse Document Frequency)

TF-IDF improves Bag of Words by assigning importance scores to words.

TF-IDF= TF*IDF

TF(t,d)= Total number of terms in document d/Number of times term t appears in document d

IDF(t)=log( Number of documents containing term t/Total number of documents)

Intuition:

Words that occur frequently in a document are important.
Words that occur in many documents are less important.

Components:

Term Frequency (TF): frequency of a word in a document

Inverse Document Frequency (IDF): rarity of the word across documents

Why TF-IDF is better than BoW:

Reduces importance of common words like the and is
Highlights meaningful words

Performs well in:

Search engines
Spam detection
Document similarity tasks

Limitations:

Does not capture semantic meaning
Synonyms are treated as different words

4. Word2Vec

Word2Vec represents words as dense numerical vectors that capture meaning and context.

Key idea:

Words used in similar contexts have similar meanings. The vectors used to represent King, Queen, Man and Woman when undergo arithmetic operations, give results as below.

Famous examples:

King − Man + Woman ≈ Queen

Paris − France + Italy ≈ Rome

Word2Vec has 2 components:

1. CBOW (Continuous Bag of Words)

Predicts a word using surrounding context.

Sentence: "Raj went to school yesterday"
Window size: 1

Input: [Raj, to] → Output: went
Input: [went, school] → Output: to
Input: [to, yesterday] → Output: school

Working:

The context words are converted to one-hot vectors.
These vectors are summed or averaged
They are passed through the hidden layer.
The model predicts the target word
Error is calculated
Weights are updated using backpropagation

2. Skip-Gram

Predicts surrounding words using a target word.
For same sentence,

Target word: went
Context words: Raj, to

Training pairs:
Input: went → Output: Raj
Input: went → Output: to

Target = to
Context words: went, school

Training pairs:
Input: to → Output: went
Input: to → Output: school

Target = school
Context words: to, yesterday

Training pairs:
Input: school → Output: to
Input: school → Output: yesterday

Working

The target word is converted to a one-hot vector
Passed through the hidden layer
The model predicts each context word
Error is calculated
Weights are updated using backpropagation

👉 The hidden layer weights become the word embeddings

Advantages:

Captures semantic relationships
Produces dense and meaningful embeddings
Useful for clustering and similarity tasks

Limitation:

Same word has the same vector in all contexts

Example:
bank (river) and bank (money)

This limitation is addressed by contextual models like BERT.

When to Use Each Technique?

Use Bag of Words when:

Building simple text classifiers
Creating baseline NLP models

Use TF-IDF when:

Working on search systems
Performing document similarity
Detecting spam

Use Word2Vec when:

Semantic similarity is important
Building recommendation systems
Clustering text data

Final Thoughts

These techniques show the evolution of NLP from counting words to weighting word importance to understanding semantic meaning.
They form the foundation for modern NLP and Generative AI systems.

🌱 NLP for Beginners: Understanding the Basics of Natural Language Processing

Ananya S — Thu, 22 Jan 2026 16:03:32 +0000

Natural Language Processing (NLP) is one of the most exciting areas of Artificial Intelligence today. From chatbots and search engines to spam detection and sentiment analysis, NLP helps machines understand human language.

If you’re just starting out and feel confused by terms like tokenization or lemmatization, this post will give you a clear and gentle introduction.

📌 What is Natural Language Processing (NLP)?

Natural Language Processing (NLP) is a subfield of Artificial Intelligence that enables computers to understand, analyze, and generate human language.

In simple terms:

NLP allows machines to work with text and speech in a meaningful way.

Real-world applications of NLP

Chatbots and virtual assistants
Google Search and autocomplete
Spam email detection
Sentiment analysis of reviews
Language translation

🗺️ A Beginner-Friendly Roadmap to Learn NLP

Before diving into complex models, it’s important to understand how text is processed.

A simple conceptual roadmap

Text Preprocessing
- Tokenization
- Stop words removal
- Stemming
- Lemmatization
Text Representation
- Bag of Words
- TF-IDF
- Word Embeddings
Classical NLP Tasks
- Text classification
- Sentiment analysis
- Named Entity Recognition
Advanced NLP (Later Stage)
- Transformers
- BERT
- GPT
- Large Language Models

🧹 Why Text Preprocessing is Important

Machines don’t understand language like humans do.

Example sentence: "I am learning Natural Language Processing!"

To a machine, this is just a sequence of characters.

Text preprocessing helps convert raw text into a format that machine learning models can understand.

✂️ Tokenization

Tokenization is the process of breaking text into smaller units called tokens.

Example

Sentence:

"I love learning NLP"

After tokenization:

["I", "love", "learning", "NLP"]

Types of tokenization

Word tokenization
Sentence tokenization
Subword tokenization (used in transformers)

🛑 Stop Words

Stop words are commonly used words that usually don’t add much meaning to the text.

Examples:

is, am, are, the, a, an, in, on, and

Why remove stop words?

They add noise
They increase dimensionality
They often don’t help in tasks like classification

🌿 Stemming

Stemming reduces words to their root form by removing suffixes.

Fast
Not always linguistically correct

Common stemming algorithms:

PorterStemmer() : just removes suffix or prefix without context understanding.
SnowballStemmer() : better than PorterStemmer and supports many languages.
RegexStemmer() : removes prefix or suffix based on given expression to be removed.

words=['eating','eaten','eat','write','writes','history','mysterious','mystery','finally','finalised','historical']
from nltk.stem import PorterStemmer
stemming=PorterStemmer()
for word in words:
    print(word+"------>"+ stemming.stem(word))

OUTPUT:
eating------>eat
eaten------>eaten
eat------>eat
write------>write
writes------>write
history------>histori
mysterious------>mysteri
mystery------>mysteri
finally------>final
finalised------>finalis
historical------>histor

Stemming just removes prefixes or suffixes and doesn't give meaning words.

🍃 Lemmatization

Lemmatization converts words into their dictionary base form, called a lemma.
NLTK provides WordNetLemmatizer class which is a thin wrapper around the wordnet corpus

from nltk.stem import WordNetLemmatizer
## WordNet is a dictionary dataset which has words with their base form.We need to download this dataset to use WordNetLemmatizer
import nltk
nltk.download('wordnet')
lemmatizer=WordNetLemmatizer()
lemmatizer.lemmatize('going') #output: going
lemmatizer.lemmatize('going', pos='v') #ouput : go
#This lemmatize command we can add pos_tags that identify the word as verb, noun, adjective, etc. to help decide how to go to root word.
## Parts of Speech: Noun -n, Verb-v, adverb-r, adjective-a. Default pos tag is 'n'

Considers grammar and context
Produces meaningful words
More accurate but slower than stemming

⚖️ Stemming vs Lemmatization

Feature	Stemming	Lemmatization
Speed	Fast	Slower
Accuracy	Lower	Higher
Output	May not be a real word	Always a valid word
Grammar-aware	❌	✅

🧠 Final Thoughts

NLP is not magic — its structured text processing combined with machine learning.
Which is your favorite concept in NLP?
Drop a comment down below!

How AI Sees Images: A Gentle Introduction to Convolutional Neural Networks

Ananya S — Wed, 31 Dec 2025 12:15:19 +0000

If you’ve ever wondered how Instagram recognizes faces, how self-driving cars see roads, or how medical scans detect diseases, the answer often starts with one thing:

Convolutional Neural Networks (CNNs)

But don’t worry — no heavy math, no scary equations.
Let’s understand CNNs step by step, visually and intuitively.

Why Not Just Use Normal Neural Networks?

Imagine you have a 100×100 image.

That’s 10,000 pixels.

If you feed this into a normal neural network:

Every pixel connects to every neuron
Millions of parameters 😵
Overfitting happens fast
Spatial information is lost

👉 CNNs were created specifically for images to solve this problem.

👀 How Humans See vs How CNNs See

When you look at an image, you don’t see pixels.

You see:

Edges
Shapes
Patterns
Objects

CNNs try to learn exactly this hierarchy.

🧩 The Core Idea of CNNs (In One Line)

CNNs learn small patterns first (edges), then combine them to learn complex objects.

🧱 The Building Blocks of a CNN

Let’s break it down.

1️⃣ Convolution Layer — The “Feature Detector”

This is the heart of CNNs ❤️

Instead of looking at the entire image at once:

CNN uses a small filter (kernel) like 3×3
Slides it across the image
Detects patterns

Example patterns:

Vertical edges
Horizontal edges
Curves
Corners

📌 Think of it like a magnifying glass scanning the image.

🔍 What is a Filter (Kernel)?

A filter is a small matrix that:

Multiplies with image pixels
Extracts specific features

Different filters learn different features automatically.

2️⃣ ReLU — Adding Non-Linearity

After convolution, we apply ReLU:

ReLU(x) = max(0, x)

Why?

Keeps positive values
Removes unnecessary noise
Makes the network capable of learning complex patterns

📌 Without ReLU, CNNs would just be fancy linear models.

3️⃣ Pooling Layer — Reducing Size, Keeping Meaning

Pooling helps:

Reduce image size
Reduce computation
Prevent overfitting

Most common:
👉 Max Pooling (2×2)

It keeps the most important value from a region.

📌 Think of it as compressing an image without losing important details.

4️⃣ Fully Connected Layer — The Decision Maker

After convolution + pooling:

We flatten everything
Feed it into a normal neural network
Make final predictions

Example outputs:

Cat / Dog
Healthy / Diseased
Car / Pedestrian / Tree

🧠 How CNNs Learn (Training Intuition)

CNNs learn by:

Making predictions
Calculating error (loss)
Adjusting filters using backpropagation
Repeating until accuracy improves

📌 CNNs don’t start smart — they become smart through data.

🧪 A Simple CNN Example (Keras)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential([
    Conv2D(32, (3,3), activation='relu', input_shape=(100,100,3)),
    MaxPooling2D(2,2),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

📌 Don’t worry about memorizing this — understand what each layer does.

🏗 How CNNs Learn Features (Very Important)

Layer Depth	Learns
Early layers	Edges, lines
Middle layers	Shapes, textures
Deep layers	Objects, faces

This is why CNNs are so powerful.

🚀 Real-World Applications of CNNs

🏥 Medical image diagnosis
🚗 Autonomous driving
📸 Face recognition
🛒 Product image search
🔍 OCR (text from images)

If it involves images or vision, CNNs are probably involved.

❌ Common Beginner Mistakes

Training huge CNNs on small datasets
Ignoring overfitting
Not visualizing results
Blindly copying architectures

📌 Start small, then scale.

🧭 Where Should You Go Next?

If you’re learning CNNs:

Build a simple image classifier
Experiment with:
- Filter sizes
- Pooling
- Dropout
Visualize predictions
Try transfer learning (ResNet, VGG)

💬 Let’s Discuss!

What confused you most about CNNs?
What project are you building with CNNs?
Want a post on CNN interview questions or visual intuition?

👇 Drop a comment — let’s learn together!

Machine Learning for Everyone: A Friendly Intro to the Future!

Ananya S — Tue, 23 Dec 2025 16:19:15 +0000

Hey dev.io fam! Ever feel like "machine learning" is this magical, super-complex term that only rocket scientists understand? Well, guess what? It's not! Machine learning is actually all around us, making our lives easier and more interesting every single day. Think of this as your friendly, no-jargon introduction to the world of ML. Let's dive in!

So, What Even Is Machine Learning?

At its core, machine learning is about teaching computers to learn from data, just like humans learn from experience. Instead of explicitly programming every single rule, we feed the computer tons of information, and it figures out the patterns and relationships on its own.
Imagine you want a computer to tell the difference between a picture of a cat and a dog. The old way would be to write down rules: "If it has pointy ears, it's a cat. If it has floppy ears, it's a dog." But what about cats with floppy ears or dogs with pointy ones? It gets complicated fast!
With machine learning, you show the computer thousands of pictures labeled "cat" and "dog." The ML model then learns what characteristics distinguish a cat from a dog.

Where Do We See ML in Action?
You're probably interacting with ML without even realizing it!

Netflix/Spotify Recommendations: Ever wonder how they know exactly what you want to watch or listen to next? Yep, that's ML working its magic, analyzing your past preferences and what similar users enjoy.
Spam Filters: Those pesky spam emails that never make it to your inbox? Thank machine learning for identifying and filtering them out.
Voice Assistants (Siri, Alexa, Google Assistant): When you ask your device a question, ML is behind the scenes, understanding your speech and figuring out the best response.
Facial Recognition on Your Phone: Unlocking your phone with your face? ML!
Online Shopping Product Suggestions: "Customers who bought this also bought..." - another classic ML example.

The Basic Ingredients: Data, Model, and Learning!

To make an ML system work, you generally need three things:

Data: This is the fuel for your ML engine. The better quality, relevant data you have, the better your model will learn.

Model: This is the algorithm or mathematical structure that learns the patterns from your data. Think of it as the "brain" that processes the information.

Learning Algorithm: This is the process that adjusts the model's parameters based on the data, minimizing errors and improving its performance.

Below is an example of house prediction using supervised learning method of Linear Regression.

from sklearn.linear_model import LinearRegression

# Features: [Square Footage] (Needs to be a 2D array)
X = [[1000], [1500], [2000], [2500]]

# Labels: Price in Dollars
y = [200000, 300000, 400000, 500000]

# Create and train the regressor
model = LinearRegression()
model.fit(X, y)

# Predict: Price for a 1800 sq ft house
price_prediction = model.predict([[1800]])

print(f"Predicted Price: ${price_prediction[0]:,.2f}")
# Output: Predicted Price: $360,000.00

The Different Flavors of ML

There are generally three main types of machine learning:

Supervised Learning: This is what we just did! We have data with both inputs (movie genre, rating) and the correct outputs (liked/didn't like). The model learns to map inputs to outputs. Think of it like learning with a teacher.
Examples: Image classification, spam detection, predicting house prices.

Unsupervised Learning: Here, the data doesn't have labels. The model tries to find hidden patterns or groupings within the data on its own. Think of it like exploring without a map.
Examples: Customer segmentation (grouping similar customers), anomaly detection (finding unusual data points), data compression.

Reinforcement Learning: This is about an agent learning to make decisions by performing actions in an environment to maximize a reward. Think of teaching a dog tricks with treats.
Examples: Training robots, game AI (like AlphaGo), self-driving cars.

Ready to Explore Further?

This is just the tip of the iceberg! The field of machine learning is constantly evolving and incredibly exciting. Don't be intimidated; start small, build some projects, and you'll be amazed at what you can create!

What are your thoughts on ML? Have you built anything cool with it? Let us know in the comments below!

Understanding Errors in Machine Learning: Accuracy, Precision, Recall & F1 Score

Ananya S — Tue, 16 Dec 2025 18:59:02 +0000

Machine Learning models are often judged by numbers, but many beginners (and even practitioners) misunderstand what those numbers actually mean. A model showing 95% accuracy might still be useless in real-world scenarios.

In this post, we’ll break down:

Types of errors in Machine Learning
Confusion Matrix
Accuracy
Precision
Recall
F1 Score

All explained intuitively, with examples you can confidently explain in interviews or apply in projects.

1️⃣ Types of Errors in Machine Learning

In a classification problem, predictions fall into four categories:

Actual \ Predicted	Positive	Negative
Positive	True Positive (TP)	False Negative (FN)
Negative	False Positive (FP)	True Negative (TN)

🔴 False Positive (Type I Error)

Model predicts Positive, but the actual result is Negative
Example: Email marked as Spam but it is actually Not Spam

🔵 False Negative (Type II Error)

Model predicts Negative, but the actual result is Positive
Example: Medical test says No Disease but the patient actually has it

These errors directly impact evaluation metrics.

2️⃣ Confusion Matrix (The Foundation)

A confusion matrix summarizes prediction results:

                Predicted
               P       N
Actual P      TP      FN
Actual N      FP      TN

All metrics are derived from this table.

3️⃣ Accuracy

📌 Definition

Accuracy measures how often the model is correct.

📐 Formula

Accuracy = (TP + TN) / (TP + FP + FN + TN)

❗ Problem with Accuracy

Accuracy can be misleading in imbalanced datasets.

Example:

99 normal patients
1 patient with disease
Model predicts No Disease for everyone

Accuracy = 99%, but the model is dangerous.

👉 Accuracy alone is not enough.

4️⃣ Precision

📌 Definition

Precision answers:

Of all predicted positives, how many are actually positive?

📐 Formula

Precision = TP / (TP + FP)

🎯 When to focus on Precision?

When False Positives are costly.

Examples:

Spam detection
Fraud detection

You don’t want to wrongly flag legitimate cases.

5️⃣ Recall (Sensitivity)

📌 Definition

Recall answers:

Of all actual positives, how many did the model correctly identify?

📐 Formula

Recall = TP / (TP + FN)

🎯 When to focus on Recall?

When False Negatives are dangerous.

Examples:

Cancer detection
Accident detection

Missing a positive case can have severe consequences.

6️⃣ Precision vs Recall Tradeoff

Increasing Precision often decreases Recall, and vice versa.

Scenario	Priority
Spam Filter	Precision
Disease Detection	Recall
Fraud Detection	Recall

This tradeoff leads us to F1 Score.

7️⃣ F1 Score

📌 Definition

F1 Score is the harmonic mean of Precision and Recall.

📐 Formula

F1 = 2 × (Precision × Recall) / (Precision + Recall)

✅ Why F1 Score?

Balances Precision & Recall
Best for imbalanced datasets
Penalizes extreme values

If either Precision or Recall is low, F1 score drops sharply.

8️⃣ Summary Table

Metric	Best Used When	Focus
Accuracy	Balanced data	Overall correctness
Precision	FP costly	Prediction quality
Recall	FN costly	Detection completeness
F1 Score	Imbalanced data	Balanced performance

9️⃣ Real-World Case Studies

Understanding metrics becomes much clearer when we map them to real-world problems. Below are some common and interview-relevant case studies.

🏥 Case Study 1: Disease Detection (Cancer / COVID)

Scenario:
A model predicts whether a patient has a disease.

Critical Error: False Negative (FN)

Predicting Healthy when the patient actually has the disease

Why Recall Matters More:

Missing a sick patient can delay treatment and cost lives
It is acceptable to have some false alarms (FPs), but not to miss real cases

✅ Primary Metric: Recall

In healthcare, we prioritize recall over precision.

💳 Case Study 2: Credit Card Fraud Detection

Scenario:
The model identifies fraudulent transactions.

Critical Error: False Negative (FN)

Fraud transaction marked as legitimate

Tradeoff:

Too many false positives annoy customers.
Too many false negatives cause financial loss.

Here if we optimize only for Recall then even the slightest unusual transaction would be considered fraud. If we use Precision, then it would only detect fraud if it were very confident.

✅ Best Metric: F1 Score

F1 balances customer experience and fraud prevention.

📧 Case Study 3: Spam Email Detection

Scenario:
Classifying emails as Spam or Not Spam.

Critical Error: False Positive (FP)

Important email marked as spam

Why Precision Matters:

Users may miss critical emails (job offers, OTPs, invoices)

✅ Primary Metric: Precision

🚗 Case Study 4: Autonomous Driving (Pedestrian Detection)

Scenario:
Detecting pedestrians using camera and sensor data.

Critical Error: False Negative (FN)

Pedestrian not detected

Why Recall is Crucial:

Missing even one pedestrian can be fatal

✅ Primary Metric: Recall

🏭 Case Study 5: Manufacturing Defect Detection

Scenario:
Detecting defective products on an assembly line.

Critical Error Depends On:

High FP → Waste and increased cost
High FN → Faulty product reaches customer

Balanced Approach:

Use Precision + Recall together

✅ Best Metric: F1 Score

🔚 Final Thoughts

Never blindly trust accuracy.
Always ask:

What kind of error is more dangerous?
Is my dataset imbalanced?
What is the real-world cost of FP vs FN?

Understanding these metrics makes you a better ML engineer, not just a model builder.

If this helped you, feel free to share or comment your favorite ML pitfall!!

The Science of Prompt Evaluation: From BLEU & ROUGE to Real Human Feedback

Ananya S — Mon, 08 Dec 2025 17:13:29 +0000

Prompt engineering feels magical—change a few words and the model behaves differently.
But how do you measure whether one prompt is actually better than another?
Just reading the outputs is not enough.
For real AI applications, you need evaluation metrics.

1. BLEU Score

BLEU (Bilingual Evaluation Understudy) checks how many n-grams from the reference appear in the generated output.
An n-gram is a sequence of n consecutive words (or tokens) in a sentence.
Unigram: “the”, “cat”, “sat”
Bigram: “the cat”, “cat sat”

Great for:

summarization
translation
structured responses

Not great for:

creative writing
open-ended generation
varied response styles

Let's say the reference is:

The cat sits on the mat.

And the LLM output is:

A cat is sitting on the mat.

BLEU will give a high score because many words and n-grams match.

2. ROUGE Score

ROUGE is more recall-focused than BLEU. ROUGE measures how much of the reference (ground-truth) text appears in the generated output.

ROUGE Recall = (overlapping units) / (total units in reference)
This is why ROUGE is used heavily in summarization, where the model must capture key points from the original text.

ROUGE-1: unigram overlap (Measures overlap of single words.)
ROUGE-2: bigram overlap (Measures overlap of word pairs.)
ROUGE-L: longest common subsequence (Measures longest in-order word sequence shared by both texts.)

Best for:

summarization
Q&A where key facts must be present
comparing prompt improvements

Example:

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)

reference = "The cat sits on the mat"
generated = "The mat has a cat sitting on it"

scores = scorer.score(reference, generated)
print(scores)

use_stemmer increases the rouge score as it ensures rouge compares words to their base meaning. Hence sitting, sits, sat will be seen similar to sit. This shows the generated text captures the meaning accurately and hence the score can be higher.

Output:

{
 'rouge1': Score(precision=0.625, recall=0.7142857142857143, fmeasure=0.6666666666666666),
 'rouge2': Score(precision=0.2727272727272727, recall=0.3333333333333333, fmeasure=0.3),
 'rougeL': Score(precision=0.5, recall=0.5555555555555556, fmeasure=0.5263157894736842)
}

Your exact numbers might differ slightly. fmeasure = F1 score
These evaluation techniques fall under Natural Language Processing (NLP) and is commonly used to assess the quality of text generation models such as chatbots, summarizers, and translation systems.
To run the example code, make sure to install the rouge_score library first:

pip install rouge_score

ROUGE-L is especially useful because it checks sentence structure similarity.

For prompt performance, you care about:

correctness
coherence
reasoning
readability
hallucinations

→ BLEU/ROUGE cannot measure these.

That’s where Human Evaluation comes in.

3. Human Evaluation

Human eval means real people rate outputs based on criteria like:

Metric	What It Measures
Accuracy	Is the answer correct?
Relevance	Is it on topic?
Clarity	Easy to read and understand?
Completeness	Did it answer the full question?
Safety	No harmful or biased content?

Simple Human Evaluation Procedure

Pick a prompt (Prompt A)
Generate output
Modify prompt → (Prompt B)
Generate output
Ask 3–5 evaluators to rate on a 1–5 scale.

Comparing Two Prompts

Prompt A

Summarize this article in one sentence.

Prompt B

Provide a short, factual one-sentence summary of the article below.

Let's say, after generating summaries for 50 samples we get:

Metric	Prompt A	Prompt B
BLEU	0.42	0.57
ROUGE-L	0.61	0.72
Human Score	3.4	4.5

→ Prompt B is clearly superior.

Conclusion

Evaluating prompt quality shouldn't be based on guesswork. Use metrics like:

BLEU → precision of n-gram match
ROUGE → recall and similarity
Human Evaluation → truth, clarity, completeness

Together, they give you a reliable, real-world view of how good your prompts actually are.

If you found this useful, hit the ❤️ or 🔥 so others can discover it too.
Which evaluation metric do you prefer—BLEU, ROUGE, or human evaluation?
To understand the meaning of precision, recall and accuracy check out this article: Understanding Errors in Machine Learning: Accuracy, Precision, Recall & F1 Score
Drop your answer in the comments. I reply to every comment!