Forem: Shravya K

How RAG Changed the Way We Use Large Language Models

Shravya K — Tue, 13 Jan 2026 08:22:22 +0000

#From guessing to grounded answers

We've been asking AI the wrong questions.

Or rather, we've been asking the right questions to the wrong system.

Large Language Models are incredible at reasoning, connecting ideas, and explaining complex concepts. But there's something they're fundamentally not designed to do: remember specific facts with perfect accuracy.

For years, we tried to make LLMs better at memorization. Bigger models, more training data, longer context windows.

Then someone had a different idea: What if we stopped trying to make them remember everything and instead taught them how to look things up?

That shift from memory to retrieval is what RAG is all about.

LLMs Don't Know. They Predict.
Here's the fundamental thing about Large Language Models: they don't look things up. They predict.

When you type a message, it gets broken into tokens (pieces of words), which flow through a neural network asking one question billions of times: "What comes next?"

This is called autoregressive generation - each new word makes sense because of all the previous ones. It's like autocomplete on steroids, predicting one token at a time based on patterns learned during training.

Prompt: 

The weather today is

What the model does
Predicts the most likely next word:

The weather today is → "sunny"

Run the same prompt twice? You'll get different answers. The model is making probabilistic choices, not retrieving stored facts.

The Models You're Actually Using

Different LLMs have different strengths:

ChatGPT (GPT-4): Best for conversations, coding, and following complex instructions
Claude: Excels at long documents and detailed analysis
Gemini: Strong with multimodal tasks (text, images, video)
Llama: Open-source, customizable for specialized applications

They all work the same way: predicting tokens based on context. And they all share the same core limitation.

The Limitation That Was Always There

LLMs can't look things up. When you ask about a certain product, the model doesn't search files. It generates an answer based on patterns from training data, data that's months old and has never seen your internal documents.

If it doesn't know something, it doesn't say "I don't know." It predicts what a reasonable answer might sound like.

The problem:

Training data becomes outdated immediately
No access to private or recent information
No verification mechanism
Confident responses regardless of accuracy

This works fine for creative tasks or explaining concepts. But for anything requiring factual accuracy? It's a dealbreaker.

The Breakthrough: Stop Memorizing, Start Retrieving

The solution came from rethinking the problem: What if LLMs don't need to remember everything?

That's RAG - Retrieval-Augmented Generation.

The process is simple:

User asks a question
System searches a knowledge base
Retrieved information gets added to the prompt
LLM reads and responds

RAG = LLM + Retrieved Context

Just like humans don't rely purely on memory when writing reports, we gather sources first, RAG lets AI read before responding. The LLM handles reasoning, retrieval handles facts.

How RAG Works: The Complete Flow

Setup Phase (happens once):

Ingestion: Load your documents (PDFs, web pages, databases)

 Gather 600 help articles from a travel booking app, 150 PDF guides about cancellations and refunds, and fare rules from the internal system.

Chunking: Break them into smaller pieces (usually a few paragraphs)

A long document explaining ticket cancellations is split into smaller sections—one for refunds, one for rescheduling, one for penalties.

Embedding: Convert each chunk into a mathematical representation

A line like "Tickets cancelled within 24 hours are fully refundable" is turned into a format that captures its meaning.

Storage: Save in a vector database optimized for similarity search

All sections are stored with tags like category="refunds", travel_type="flight", and last_updated="2024".

Query Phase (every time someone asks):

User question: "My flight booking disappeared even though I paid for it"

Convert the question into a mathematical form

[booking_issue, payment_done, booking_missing, flight]

Search for chunks with similar meanings

"booking not confirmed after payment"
"payment successful but ticket not issued"
"flight booking disappeared"

Retrieve the top matches (usually 3-10 chunks)

Chunk 1:
"If payment is successful but ticket is not issued within 15 minutes,
the booking may remain in pending state."

Chunk 2:
"Bookings are auto-cancelled if airline confirmation is not received."

Chunk 3:
"Some banks show payment success even when airline rejects the transaction."

Chunk 4:
"Pending bookings disappear from user dashboard after 30 minutes."

Chunk 5:
"Refunds for failed bookings are processed within 5–7 days."

Insert these chunks into the prompt sent to the LLM


Context:
- Payment success does not always mean ticket issued
- Pending bookings may disappear after 30 minutes
- Airline confirmation failure causes auto-cancellation
- Refund timeline: 5–7 days

Answer the user clearly.

LLM reads and generates an answer based on retrieved context

FInal answer:
“Your booking disappeared because the airline did not confirm the ticket after payment. The system auto-cancelled it, and your payment will be refunded within 5–7 days.”

The LLM never sees your entire document collection — just the most relevant pieces for each specific question.

Three Ways to Find Information

Retrieval isn't just "search" — there are three fundamentally different approaches:

1. Keyword Search: Exact Matching

Finds documents with specific words or phrases. Fast and predictable, but misses synonyms.
If the words don’t match, it pretends the information doesn’t exist.

Example:
Search: “pizza menu”
 Finds: “Pizza Menu – Downtown Branch”
 Misses: “Our Italian Dishes” or “What We Serve”

2. Semantic Search: Understanding Meaning

Uses mathematical representations (embeddings) to find conceptually similar content, even with different wording.

Search: “I’m hungry but don’t want to cook”
 Finds: “Nearby restaurants,” “Food delivery options,” “Quick meals”

3. Metadata Filtering: Structured Criteria

Applies hard rules: Metadata filtering narrows results using clear rules and labels.

Search: “Show my assignments”
 Filters: subject = “AI” AND status = “pending” AND due_date = “this week”

Why Use All Three (Hybrid Search)

Modern RAG systems combine these approaches:

Semantic search finds conceptually relevant content
Keyword search ensures exact matches aren't missed
Metadata filtering applies necessary constraints

Each catches what the others miss.

Real-World Example

Scenario: Food Ordering App Help Chatbot

Setup:

Knowledge base:

300 FAQs (orders, payments, delivery)
150 issue guides (refunds, late delivery, app bugs)

Process:

Break articles into small pieces
Turn them into searchable representations
Store them so the system can quickly look them up

When user asks: "Why does my food order get cancelled every time I add desserts?"

Converts the question into a form it can compare with stored information

Looks for related content using:

Meaning (order issues, add-ons, desserts)
Keywords (“cancelled,” “desserts,” “order”)

Applies filters:

category = “order problems”
status = “current”

Picks the top 5 most relevant explanations and adds them to the prompt

Result: LLM response is based on actual app rules, not guesswork.

Before RAG: "Please try restarting the app or placing the order again." (generic)

With RAG: "Orders with desserts get cancelled if the restaurant is marked ‘no cold storage’. Try removing desserts or choosing a different outlet." (specific, accurate)

Why This Changes Everything

RAG enables use cases that weren't reliably possible before:

Internal company AI: Train on your specific docs, policies, codebase
Medical assistance: Reference current treatment guidelines and research
Legal research: Search case law with specific citations
Customer support: Know exact product features and current policies
Research tools: Find and connect recent publications

All of these need one thing: answers grounded in specific, current, trusted information.

The Core Insight

Large Language Models are reasoning engines that happen to be trained on data. We mistook that training data for a feature when it was really a limitation.

RAG separates reasoning from knowledge:

LLM: Understanding context, connecting ideas, generating responses
Retrieval: Storing information, keeping it current, finding relevant pieces

This division of labor makes AI systems reliable.

The future isn't about models that know everything. It's about systems that know how to find what they need, when they need it, from trusted sources.

That's what RAG represents. And it's just the beginning.

I Thought I Knew How To Talk To AI: I Didn't

Shravya K — Mon, 12 Jan 2026 13:01:42 +0000

The first time I asked ChatGPT for help, I typed: "Write me a product description for noise-canceling headphones."
It gave me garbage.

I blamed the AI. But the problem was me.

"Be Clear" Doesn't Mean "Be Long"

For months, I wrote these rambling prompts explaining every little detail.
Results were still hit-or-miss.

Then I learned something simple:

DELIMITERS

They're the lines you draw in your prompt to say, "This part is my instruction. This part is the content I want you to work with. Don't confuse the two".
Think of it like writing on a whiteboard during a meeting. You don't just scribble everything together in one giant blob. You draw boxes. You underline headers. You separate the problem from the solution.
Delimiters do the same thing, they keep your instructions from bleeding into your content, so the AI knows exactly what's what.

Eg: Triple quotes """, XML tags , or even simple dashes ---— pick whatever feels natural.

Suddenly, the AI knew exactly what I wanted.

Role:
You are an AI writing assistant for students.
Task:
Improve clarity; keep it short; avoid complex words.
Rules:
- Do not change meaning
- Output only the improved text
- No extra explanations
Input | Metadata:
topic: AI prompting | level: beginner | tone: casual
---
Original text:
"""
AI works good but sometimes it gives wrong answers because the question is not clear.
"""
---
Output format:
improved_text: <one improved sentence>

Stop Rushing the AI (And Yourself)

I used to treat AI like a vending machine.
Input -> enter -> expect perfection.

Then someone told me: give the model time to think.
Not literal time. Structure.

Instead of "Solve this," I learned to say:

"First, analyze the problem" - Understand what you're dealing with

A train travels 60 km in 1.5 hours.
The speed of the train is constant.
We are asked to find the time required to travel 100 km at the same speed.

So this is a speed–distance–time problem.

"Then, identify approaches" - Explore different ways forward

We can solve this by:
Calculating the speed of the train, then
Using the same speed to find the time for 100 km

Formulae involved:
Speed = Distance ÷ Time
Time = Distance ÷ Speed

"Finally, provide your solution" - Deliver the answer that fits best

Calculate speed:

Speed = 60 km ÷ 1.5 hours
Speed = 40 km/h
Now calculate time for 100 km:
Time = 100 km ÷ 40 km/h
Time = 2.5 hours

The difference was massive.

Four Ways to Ask (That Actually Work)

Zero-shot: Just the instruction. No examples.
Works when the task is straightforward.

Prompt: 
Roast my coding skills.

Output: Your code works, but even it looks surprised by that.

Few-shot: Multiple examples showing a clear pattern.
My go-to for consistency.

Prompt:
Roast my coding skills.

Examples:
"Your variable names are so creative, even you forget what they mean."
"Your code has comments because future-you will be very confused."

Now roast me.

Output: Your code is like a mystery novel - long, confusing, and the ending barely makes sense

Context-based: Background + constraints + question.
Changed everything.

Prompt:
Context:
I’m a student who codes regularly.
This roast can be harsh.
Tone should be playful and relatable.

Task:
Roast my coding skills in one line.

Output: Your code works, but only because the bugs are too tired to fight back

Same question. Completely different answer.

What Finally Made It Click

I stopped thinking of prompting as "giving commands."
I started thinking of it as talking to someone who wants to help but needs to know what you actually need.
You wouldn't tell a colleague "Do the thing."
You'd give context, constraints, examples.
That's all a good prompt is.

Start Here

Use delimiters.
Separate your instructions from your content.

That alone will transform your results.
The rest will come naturally once you realize AI isn't a tool you command, it's a collaborator you communicate with.
And the quality of what you build together depends on how clearly you communicate what you're trying to build.