Forem: Nrk Raju Guthikonda

Your Voice Assistant Doesn't Need the Cloud — Here's How I Built 5 Offline NLP Tools

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:35:15 +0000

Every time I build an AI-powered tool that requires an internet connection, I feel a small pang of guilt. We've normalized shipping software that stops working the moment a cloud API goes down, a subscription lapses, or a user happens to be on an airplane. But here's the thing: most NLP tasks — sentiment analysis, text summarization, conversational AI, even voice assistants — don't need the cloud anymore.

Over the past year, I've built a series of open-source tools that prove this point. They handle voice calls, language tutoring, sentiment dashboards, news digestion, and research paper Q&A — all running locally with Ollama and models like Gemma 4. No API keys. No cloud bills. No data leaving your machine.

In this post, I'll walk through the patterns I've found most effective for building offline-first NLP applications in Python, with real code from five of my projects.

Why Offline NLP Matters More Than You Think

The conversation around AI tooling is dominated by cloud-first thinking. GPT-4o, Claude, Gemini — they're brilliant, but they come with strings attached:

Privacy: Every prompt you send is processed on someone else's server. For healthcare data, legal documents, or personal conversations, that's a non-starter.
Cost: API calls add up fast. A sentiment analysis pipeline processing 10,000 documents a day can cost hundreds per month.
Reliability: Cloud APIs have rate limits, outages, and deprecation cycles. Your local GPU doesn't.
Latency: A local model on an M-series Mac or a decent NVIDIA card returns responses in milliseconds, not seconds.

In my experience building search and retrieval systems, I've learned that the best AI tool is the one that's always available. Local LLMs make that possible for a surprising range of NLP tasks.

The Foundation: Ollama as Your Local AI Runtime

Every project I'll discuss uses the same foundation: Ollama running a local model (typically Gemma 4). The setup is dead simple:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull gemma3:4b

# Verify it's running
ollama list

And the Python integration pattern I use across all my projects:

import requests
import json

def query_local_llm(prompt: str, model: str = "gemma3:4b") -> str:
    """Send a prompt to the local Ollama instance and return the response."""
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": prompt,
            "stream": False,
            "options": {
                "temperature": 0.7,
                "top_p": 0.9,
                "num_predict": 1024,
            }
        }
    )
    response.raise_for_status()
    return response.json()["response"]

This function is the beating heart of every tool I build. It's simple, it's reliable, and it works identically whether you're on a MacBook, a Linux workstation, or a Windows machine with WSL.

Project 1: CallPilot — A Voice AI Assistant

CallPilot is probably the most ambitious project in this collection. It's an AI-powered outbound phone call assistant: you give it a phone number and instructions ("Book a dentist appointment for Tuesday at 3pm"), and it handles the entire conversation.

The architecture bridges Twilio's real-time voice streaming with an AI backend, using RAG (Retrieval-Augmented Generation) with ChromaDB to give the AI access to personal documents like insurance cards or medical records during calls.

from chromadb import Client
from chromadb.config import Settings

def build_context(query: str, collection_name: str = "documents") -> str:
    """Retrieve relevant context from local vector store for RAG."""
    client = Client(Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory="./vectorstore"
    ))
    collection = client.get_collection(collection_name)
    results = collection.query(
        query_texts=[query],
        n_results=5
    )
    chunks = results["documents"][0]
    return "\n\n".join(chunks)

The key insight here is that voice AI doesn't have to mean "send everything to a cloud transcription service." The RAG pipeline runs entirely locally — your documents are chunked, embedded, and stored in a local ChromaDB instance. When the AI needs context during a call, it queries the vector store on your machine.

While CallPilot currently uses OpenAI's Realtime API for the voice streaming component (real-time bidirectional audio is still a hard problem for local models), the entire knowledge retrieval pipeline is local. As local speech-to-text and text-to-speech models improve, the goal is to make this fully offline.

Project 2: Language Learning Bot — Conversational AI for Education

Language Learning Bot is a polyglot companion that supports 15 languages through conversation practice, vocabulary drills, and structured lessons — all powered by a local LLM via Ollama.

The conversation engine adapts to beginner, intermediate, or advanced levels and provides real-time corrections with grammar explanations:

def create_tutor_prompt(language: str, level: str, user_message: str) -> str:
    """Build a language tutor system prompt for the local LLM."""
    return f"""You are a friendly {language} language tutor.
The student's level is {level}.

Rules:
- Respond primarily in {language} with English translations in parentheses
- Correct any grammar mistakes gently, explaining the rule
- Adapt vocabulary complexity to {level} level
- Include cultural context when relevant
- End each response with a follow-up question to keep practicing

Student says: {user_message}"""

# Usage with local Ollama
response = query_local_llm(
    create_tutor_prompt("Spanish", "beginner", "Yo quiero ir al parque")
)

What makes this project compelling for offline use is the privacy angle. Language learners make mistakes — that's the whole point. Having those mistakes processed locally, never logged on a remote server, creates a psychologically safer learning environment. Every chat session, vocabulary list, and progress metric stays on the user's machine in a local JSON store.

Project 3: Sentiment Analysis Dashboard — Text Analytics with Streamlit

Sentiment Analysis Dashboard processes text files through an LLM-powered classification pipeline with confidence scores, trend detection, and word cloud generation.

The core analysis pattern uses structured prompting to get consistent, parseable output from the local LLM:

import json

def analyze_sentiment(text: str) -> dict:
    """Analyze sentiment of text using the local LLM with structured output."""
    prompt = f"""Analyze the sentiment of the following text.
Return ONLY a JSON object with these fields:
- sentiment: one of "positive", "negative", "neutral", "mixed"
- confidence: float between 0.0 and 1.0
- key_phrases: list of 3-5 important phrases from the text
- summary: one-sentence summary of the overall tone

Text: {text}

JSON:"""

    raw_response = query_local_llm(prompt)
    return json.loads(raw_response.strip())

The Streamlit dashboard renders these results into interactive visualizations — sentiment distribution charts, sliding-window trend analysis, and word clouds. The entire pipeline processes text at seconds-per-entry with batch support, compared to minutes per entry for manual review.

What I find most valuable here is the consistency. A human reviewer's sentiment judgment drifts throughout the day based on fatigue and mood. The local LLM produces consistent classifications with quantified confidence scores, and it does it without sending your text data to any third party.

Project 4: News Digest Generator — Information Triage at Scale

News Digest Generator tackles information overload. Drop a folder of .txt news articles on it and get back a structured, categorized digest with sentiment analysis and trend detection.

The categorization pipeline is where the local LLM really shines:

def categorize_articles(articles: list[dict], num_categories: int = 5) -> dict:
    """Group articles into topic categories using the local LLM."""
    titles = "\n".join(
        f"- [{i}] {a['title']}" for i, a in enumerate(articles)
    )
    prompt = f"""Given these news articles, group them into exactly
{num_categories} topic categories.

Articles:
{titles}

Return a JSON array where each element has:
- "category": short category name
- "article_indices": list of article index numbers
- "summary": 2-3 sentence summary of this topic cluster

JSON:"""

    return json.loads(query_local_llm(prompt))

The digest output includes key headlines, topic summaries, per-article sentiment, trending themes, and a forward-looking outlook section. It's the kind of tool that journalists, analysts, and researchers can run on sensitive or proprietary content without worrying about data leakage.

Project 5: Research Paper Q&A — RAG for Academic Literature

Research Paper Q&A lets you drop PDF research papers into a folder and ask questions about them in natural language. It uses a RAG pipeline with ChromaDB to chunk, embed, and retrieve relevant passages, then feeds them to Gemma 4 for answer generation.

def ask_paper(question: str, paper_chunks: list[str]) -> str:
    """Answer a question about a research paper using RAG."""
    # Retrieve the most relevant chunks
    relevant = retrieve_chunks(question, paper_chunks, top_k=5)
    context = "\n---\n".join(relevant)

    prompt = f"""Based on the following excerpts from a research paper,
answer the question. Only use information from the provided excerpts.
If the answer isn't in the excerpts, say so.

Excerpts:
{context}

Question: {question}

Answer:"""

    return query_local_llm(prompt)

This is perhaps the most natural fit for offline AI. Researchers often work with pre-publication papers, proprietary datasets, or materials under NDA. A local RAG pipeline means you can ask "What methodology did they use for the control group?" without that question — or the paper's content — ever touching an external server.

Patterns I Keep Coming Back To

After building these five projects (and many more — I'm at 116+ open-source repos now), certain patterns have proven themselves repeatedly:

Structured prompting for parseable output: Always ask the LLM to return JSON with a specific schema. It makes downstream processing predictable and testable.
Local vector stores for RAG: ChromaDB with persistent storage is lightweight enough to embed in any project. The retrieval quality with even small embedding models is excellent for domain-specific content.
Ollama as the universal runtime: By standardizing on Ollama's API, every project works with any compatible model. Swap Gemma for Llama or Mistral with a single config change.
CLI-first, web-second: Every project starts as a Click CLI tool, then gets a Streamlit or Gradio web UI. This ensures the core logic is clean, testable, and scriptable before any UI complexity enters the picture.
Privacy by architecture, not policy: When the LLM runs on localhost:11434, there's no privacy policy to read. The data physically cannot leave the machine.

Getting Started

If you want to explore any of these projects:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull gemma3:4b

# Clone any project
git clone https://github.com/kennedyraju55/sentiment-analysis-dashboard.git
cd sentiment-analysis-dashboard
pip install -r requirements.txt

# Run the CLI
python main.py analyze --file sample.txt

# Or launch the web UI
streamlit run app.py

All five projects follow the same structure: install Ollama, pull a model, clone the repo, install dependencies, and run. No API keys. No account creation. No cloud configuration.

What's Next

The local LLM ecosystem is evolving fast. Models are getting smaller and more capable. Ollama recently added vision model support, which opens up entirely new offline use cases — document OCR, image-based Q&A, multimodal assistants. I'm actively building tools that leverage these capabilities.

The thesis is simple: if your NLP tool requires an internet connection and it doesn't strictly need one, you're shipping a worse product than you could be. Local LLMs have crossed the quality threshold for production use in dozens of NLP tasks. The tools I've shared here prove it.

Every one of these projects is MIT-licensed and open source. Clone them, break them, improve them. That's the whole point.

About the Author

Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, working on semantic indexing and retrieval-augmented generation. Outside of work, he maintains 116+ open-source repositories exploring AI, NLP, healthcare tech, developer tools, and creative applications — all built with local LLMs and a privacy-first philosophy.

🐙 GitHub: github.com/kennedyraju55
✍️ Blog: dev.to/kennedyraju55
💼 LinkedIn: linkedin.com/in/nrk-raju-guthikonda-504066a8

I Built 5 AI Developer Tools That Run Entirely on My Laptop — No API Keys, No Cloud, No Limits

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:29:52 +0000

Every developer has felt the friction: you want AI to help with a mundane task — writing standup notes, reviewing a pull request, generating boilerplate — but the moment you reach for a cloud API, you hit rate limits, accumulate costs, or worse, realize you can't send proprietary code to a third-party endpoint.

What if the AI lived on your machine? No API keys. No network dependency. No billing surprises. Just a local model serving intelligent responses over localhost.

Over the past year, I've built a suite of open-source developer productivity tools that run entirely on local LLMs using Ollama and Google's Gemma model family. In this post, I'll walk through the architecture, share real code, and explain why local-first AI is the most practical path for developer tooling today.

Why Local LLMs for Developer Tools?

Cloud-hosted LLMs are powerful, but they carry trade-offs that matter in daily engineering workflows:

Cost accumulates fast. A team of ten engineers each making 50 AI-assisted queries per day burns through API credits quickly. Local inference is free after the initial model download.
Offline-first matters. Planes, coffee shops with spotty Wi-Fi, corporate VPNs that block external endpoints — local models don't care.
Privacy is non-negotiable. When you're reviewing code from a private repository or generating reports that reference internal project names, sending that context to a remote API is a risk. Local inference keeps everything on-disk.
Latency is predictable. No cold starts, no queue wait times, no variable response times based on provider load. A 4B parameter model on a modern laptop with 16 GB RAM responds in 1–3 seconds consistently.

In my experience building production search and retrieval systems, I've learned that the best developer tools are the ones with zero friction to adopt. Local LLMs eliminate the biggest friction point: setup and credentials.

The Stack: Ollama + Gemma + FastAPI

The architecture I've converged on across multiple projects is deliberately simple:

┌─────────────────────────────────────────────┐
│              Developer's Laptop              │
│                                              │
│  ┌──────────┐    HTTP     ┌──────────────┐  │
│  │  FastAPI  │ ◄────────► │   Ollama      │  │
│  │  App      │  localhost  │   (Gemma 4)   │  │
│  │  :8000    │   :11434    │   4B params   │  │
│  └──────────┘             └──────────────┘  │
│       ▲                                      │
│       │  Browser / CLI / IDE Plugin          │
│  ┌──────────┐                                │
│  │   User   │                                │
│  └──────────┘                                │
└─────────────────────────────────────────────┘

Ollama handles model management and inference. One command pulls a model, and it serves an OpenAI-compatible API on localhost:11434.

Gemma 4 (4B) is the sweet spot — small enough to run on laptops without a dedicated GPU, capable enough for code understanding, summarization, and generation tasks.

FastAPI provides the application layer: prompt engineering, input validation, structured output parsing, and a clean UI or CLI interface.

Getting Started: Ollama in 60 Seconds

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull the model (one-time ~2.5 GB download)
ollama pull gemma3:4b

# Verify it's running
curl http://localhost:11434/api/tags

On Windows, download the installer from ollama.com and Ollama runs as a background service automatically.

Project 1: AI Standup Generator

Every morning, the same ritual: open your git log, skim through Jira tickets, and type up a standup update that nobody will remember five minutes later. The standup-generator automates this entirely.

You feed it bullet points about what you worked on, and the local LLM transforms them into a structured standup report with "Yesterday," "Today," and "Blockers" sections.

import httpx

OLLAMA_URL = "http://localhost:11434/api/generate"

def generate_standup(raw_notes: str) -> str:
    prompt = f"""You are a concise engineering standup assistant.
Given these raw notes, produce a structured standup report
with sections: Yesterday, Today, Blockers.
Keep each bullet under 15 words.

Raw notes:
{raw_notes}
"""
    response = httpx.post(
        OLLAMA_URL,
        json={
            "model": "gemma3:4b",
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.3}
        },
        timeout=30.0,
    )
    return response.json()["response"]

The key design decisions:

Low temperature (0.3) keeps output deterministic — standups shouldn't be creative writing.
Stream disabled for simplicity in CLI/API mode; enable it for real-time UI feedback.
httpx over requests because it's async-friendly when you graduate to FastAPI endpoints.

Project 2: AI Code Review Bot

Code reviews are where local AI shines brightest. You absolutely should not send your team's proprietary code to a third-party API for review. The code-review-bot runs a local Gemma model to analyze diffs and surface issues.

from pathlib import Path
import httpx

def review_code(file_path: str) -> str:
    code = Path(file_path).read_text()
    prompt = f"""You are a senior code reviewer. Analyze this code for:
1. Bugs or logic errors
2. Security vulnerabilities
3. Performance concerns
4. Readability improvements

Be specific. Reference line numbers. Skip style nitpicks.

python
{code}


    response = httpx.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "gemma3:4b",
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.2, "num_ctx": 8192},
        },
        timeout=60.0,
    )
    return response.json()["response"]

Notice num_ctx: 8192 — this extends the context window so the model can ingest larger files. For a 4B model, 8K tokens is the practical ceiling before quality degrades.

Project 3: Cover Letter Generator

Job applications are tedious. The cover-letter-generator takes a job description and your resume bullets, then produces a tailored cover letter — all without sending your personal career history to OpenAI's servers.

def generate_cover_letter(
    job_description: str,
    resume_points: list[str],
    company_name: str,
) -> str:
    resume_text = "\\n".join(f"- {point}" for point in resume_points)
    prompt = f"""Write a professional cover letter for {company_name}.

Job Description:
{job_description}

Candidate's Key Qualifications:
{resume_text}

Requirements:
- 3 paragraphs maximum
- Specific connections between qualifications and job requirements
- Professional but authentic tone
- No generic filler sentences
"""
    response = httpx.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "gemma3:4b",
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.5},
        },
        timeout=45.0,
    )
    return response.json()["response"]

Temperature at 0.5 here — slightly higher than standup or code review because cover letters benefit from a touch of variability while staying professional.

Beyond AI: The Full Developer Toolkit

Not every productivity tool needs an LLM. Two other projects in my toolkit solve pure engineering problems:

apiwatch — An API contract testing and health monitoring CLI. You define API contracts in YAML, and apiwatch continuously validates your endpoints against those contracts. It catches breaking changes, performance degradation, and response schema violations before they hit production. Think of it as a lightweight Pact alternative that runs from a single CLI command.

loadlens — A load testing and capacity planning toolkit built in Python. It helps teams understand their actual throughput — including why "8 RPS per machine" might be less impressive than it sounds when you factor in connection overhead, payload size, and downstream dependencies.

Both tools follow the same philosophy: zero external dependencies for core functionality, runs anywhere Python runs, and delivers value in under five minutes of setup.

Patterns That Work Across All These Tools

After building 116+ open-source repositories, certain patterns consistently emerge:

1. Structured Prompts with Clear Constraints

The biggest improvement in local LLM output comes not from model size but from prompt structure. Always tell the model:

What role to assume
What input format to expect
What output format you need
What to exclude (often more important than what to include)

2. Temperature as a Knob, Not a Setting

Use Case	Temperature	Why
Code review	0.1–0.2	Deterministic, factual analysis
Standup reports	0.2–0.3	Structured but slightly varied phrasing
Cover letters	0.4–0.6	Natural language that doesn't sound robotic
Creative writing	0.7–0.9	Exploratory, varied output

3. Timeout Budgets

Local models on CPU can take 10–30 seconds for complex prompts. Always set explicit timeouts and provide user feedback (progress indicators or streaming responses) so the tool doesn't feel broken.

4. Graceful Degradation

def safe_generate(prompt: str, fallback: str = "") -> str:
    try:
        response = httpx.post(
            "http://localhost:11434/api/generate",
            json={"model": "gemma3:4b", "prompt": prompt, "stream": False},
            timeout=30.0,
        )
        response.raise_for_status()
        return response.json()["response"]
    except (httpx.ConnectError, httpx.TimeoutException):
        return fallback or "⚠️ Ollama is not running. Start it with: ollama serve"

If Ollama isn't running, the tool should say so — not crash with a stack trace.

What's Next: The Local AI Developer Stack

The trajectory is clear. Models are getting smaller and more capable. Gemma 4 at 4B parameters today outperforms GPT-3.5 on many code tasks. By next year, we'll likely have sub-2B models that handle most developer productivity use cases.

I'm working on expanding this toolkit to include:

Git commit message generation from staged diffs
Documentation generator that reads code and produces API docs
Test case suggester that analyzes functions and proposes edge cases

All local. All open source. All free.

Try It Yourself

Every project mentioned in this post is open source and ready to run:

Install Ollama
Pull a model: ollama pull gemma3:4b
Clone any repo and follow the README
Start building your own local AI tools

The best developer tools are the ones you control completely. When the AI runs on your machine, you own the entire stack — model, data, and output. No vendor lock-in, no usage caps, no privacy concerns.

Start local. Ship faster.

Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, working on semantic indexing and retrieval-augmented generation (RAG) systems. He maintains 116+ open-source repositories exploring AI, developer tools, healthcare technology, and creative applications of local LLMs.

🐙 GitHub: github.com/kennedyraju55
✍️ Dev.to: dev.to/kennedyraju55
💼 LinkedIn: linkedin.com/in/nrk-raju-guthikonda-504066a8

Stop Sending Your Security Alerts to Cloud AI — Build Local LLM Tools Instead

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:25:14 +0000

Every time a security analyst pastes a suspicious log entry into a cloud-based AI chatbot, they might be handing adversaries a roadmap. That firewall alert contains your internal IP ranges. That phishing email reveals which executives are being targeted. That threat intelligence report maps your entire attack surface.

I learned this the hard way. As a Senior Software Engineer at Microsoft working on Copilot Search Infrastructure, I spend my days thinking about how AI systems ingest, index, and retrieve sensitive data at scale. That experience taught me a foundational principle: the most dangerous data leak is the one disguised as a productivity tool.

So I built five open-source security AI tools — all powered by local LLMs through Ollama — that never send a single byte to the cloud. Here is why you should do the same, and how to get started.

Why Security Data Must Never Leave Your Network

This is not theoretical paranoia. It is operational reality.

1. Compliance Exposure

NIST SP 800-171, SOC 2, HIPAA, and GDPR all impose strict controls on where sensitive data can be processed. The moment you paste a security alert into a cloud AI service, you have potentially created a compliance violation. Most cloud AI providers explicitly state in their terms of service that they may use input data for model improvement.

2. Adversarial Intelligence Leakage

Security alerts are not just operational noise — they are intelligence. An alert about a brute-force attempt on admin@internal-crm.yourcompany.com tells an attacker three things: you have a CRM system, it uses that naming convention, and it is internet-facing. Sending this to a third-party API, even an encrypted one, expands your blast radius.

3. Supply Chain Risk

Cloud AI providers are themselves targets. A breach at your AI provider could expose every query ever sent — including your security telemetry. Running locally eliminates this entire attack surface.

4. Latency in Incident Response

During an active incident, you cannot afford to wait for API rate limits or deal with cloud outages. Local inference means your AI triage tools work even when the network is compromised — which is exactly when you need them most.

The Local LLM Stack: Ollama + Python

The architecture is simpler than you might expect. Ollama provides a local REST API that is compatible with the interface patterns most developers already know. Here is the foundation every tool in my security suite shares:

import requests
import json
from typing import Optional

class LocalLLM:
    """Interface to local Ollama instance for security analysis."""

    def __init__(self, model: str = "gemma4", base_url: str = "http://localhost:11434"):
        self.model = model
        self.base_url = base_url

    def analyze(self, prompt: str, temperature: float = 0.3) -> str:
        """Send a prompt to the local LLM. No data leaves localhost."""
        response = requests.post(
            f"{self.base_url}/api/generate",
            json={
                "model": self.model,
                "prompt": prompt,
                "temperature": temperature,
                "stream": False
            }
        )
        response.raise_for_status()
        return response.json()["response"]

    def health_check(self) -> bool:
        """Verify Ollama is running before processing sensitive data."""
        try:
            resp = requests.get(f"{self.base_url}/api/tags")
            return resp.status_code == 200
        except requests.ConnectionError:
            return False

Notice the low temperature setting of 0.3. For security analysis, you want deterministic, factual responses — not creative writing. This is a deliberate architectural choice that differs from most chatbot configurations.

Building a Security Alert Analyzer

Let me walk through a concrete example: triaging a cybersecurity alert. The key insight is that not everything requires an LLM. Pattern extraction (IPs, hashes, CVEs) is best handled by regex, while the LLM handles contextual analysis and summarization.

import re
from dataclasses import dataclass

@dataclass
class SecurityAlert:
    raw_text: str
    iocs: dict
    threat_score: float
    summary: str

def extract_iocs(alert_text: str) -> dict:
    """Extract Indicators of Compromise without an LLM."""
    return {
        "ips": re.findall(r'\b(?:\d{1,3}\.){3}\d{1,3}\b', alert_text),
        "domains": re.findall(r'\b[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}\b', alert_text),
        "cves": re.findall(r'CVE-\d{4}-\d{4,}', alert_text),
        "md5_hashes": re.findall(r'\b[a-fA-F0-9]{32}\b', alert_text),
        "sha256_hashes": re.findall(r'\b[a-fA-F0-9]{64}\b', alert_text),
    }

def analyze_alert(alert_text: str, llm: LocalLLM) -> SecurityAlert:
    """Full alert analysis: regex extraction + local LLM summarization."""
    iocs = extract_iocs(alert_text)

    prompt = f"""You are a senior SOC analyst. Analyze this security alert and provide:
1. Threat severity (CRITICAL/HIGH/MEDIUM/LOW)
2. Attack type classification
3. Recommended immediate actions
4. IOC summary

Alert:
{alert_text}

Extracted IOCs: {json.dumps(iocs, indent=2)}
"""
    summary = llm.analyze(prompt)

    # Score based on IOC density and keyword severity
    score = len(iocs["cves"]) * 3.0 + len(iocs["ips"]) * 1.5
    if any(kw in alert_text.lower() for kw in ["critical", "exploit", "ransomware"]):
        score += 5.0

    return SecurityAlert(
        raw_text=alert_text,
        iocs=iocs,
        threat_score=min(score, 10.0),
        summary=summary
    )

This hybrid approach — deterministic extraction plus LLM analysis — gives you the reliability of pattern matching with the contextual intelligence of a language model. And everything stays on localhost:11434.

Five Tools, Zero Cloud Dependencies

I have built and open-sourced a suite of security tools that follow this architecture. Each one solves a real problem I have encountered in production environments:

1. Cybersecurity Alert Summarizer

The flagship tool. It ingests raw security alerts, extracts IOCs (IPs, domains, hashes, CVEs), queries a local CVE database for CVSS scores, calculates weighted threat scores, and generates executive-ready summaries. The correlation engine links related alerts across multiple data sources — critical for spotting coordinated attacks.

Tech: Python, Ollama, Click CLI, FastAPI, Rich, Docker

GitHub: cybersecurity-alert-summarizer

2. DocShield — Privacy-First Document Analysis

A multi-agent system using Gemma 4 that reads, explains, and audits sensitive documents. While originally built for medical documents (HIPAA compliance demands local processing), the architecture applies to any document type containing sensitive data — contracts, financial reports, legal discovery. Five specialized agents (Orchestrator, Reader, Explainer, Checker, Bill Analyzer) work in a pipeline, each with a focused responsibility.

Tech: Python, Gemma 4, Flask, Multi-Agent Pipeline, Docker

GitHub: docshield

3. Password Strength Advisor

Goes far beyond "must contain uppercase and special character." This tool calculates Shannon entropy with pattern penalty scoring, checks against a local breach database with leet-speak variation detection, generates NIST SP 800-63B compliant policies, and creates cryptographically secure passwords using Fisher-Yates shuffling. The LLM provides natural-language explanations of why a password is weak.

Tech: Python, Ollama, Click, Streamlit, FastAPI

GitHub: password-strength-advisor

4. Phishing Email Detector

Analyzes email headers, body text, and embedded URLs to classify phishing attempts. The local LLM examines linguistic patterns (urgency cues, authority impersonation, grammatical anomalies) while deterministic checks handle SPF/DKIM validation and URL reputation lookups against local threat feeds. No email content ever leaves the analysis machine.

5. Threat Intelligence Summarizer

Ingests threat intelligence reports (STIX/TAXII feeds, vendor advisories, CVE bulletins) and produces actionable summaries for different audiences — technical IOC lists for the SOC team, risk assessments for management, patch priority lists for the infrastructure team. The LLM translates dense technical reports into audience-appropriate language.

The Architecture Pattern

Every tool in this suite follows the same layered design:

┌─────────────────────────────────────────────┐
│           Input Layer (CLI / Web / API)      │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│     Deterministic Processing Layer          │
│  (Regex, Pattern Matching, Scoring, DB)     │
│  → No LLM needed, fast, reliable            │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│     Local LLM Analysis Layer                │
│  (Ollama → Gemma 4 / Llama 3.2)            │
│  → Contextual analysis, summarization       │
│  → 127.0.0.1 only, no external calls        │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│     Output Layer (Rich CLI / Streamlit)     │
│  → Formatted reports, threat dashboards     │
└─────────────────────────────────────────────┘

The critical design decision is the separation between deterministic and LLM layers. Pattern extraction, scoring, and database lookups do not need an LLM and should not use one. The LLM handles what it is good at: contextual understanding, summarization, and natural-language generation.

Getting Started in 5 Minutes

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a security-optimized model
ollama pull gemma4

# Clone any tool from the suite
git clone https://github.com/kennedyraju55/cybersecurity-alert-summarizer.git
cd cybersecurity-alert-summarizer

# Install and run
pip install -r requirements.txt
python -m src.cyber_alert.cli --alert alerts/sample.txt

For model selection, I recommend Gemma 4 for its strong reasoning capabilities and multimodal support, or Llama 3.2 (3B) if you need faster inference on limited hardware. Both run comfortably on a machine with 16GB RAM.

The Bottom Line

Cloud AI is transformative for many use cases. Security is not one of them. The data you are analyzing — alerts, logs, threat intel, credentials, internal network topology — is precisely the data that adversaries want. Every cloud API call is an exposure surface.

Local LLMs have reached a capability threshold where they handle security analysis tasks effectively. The tools exist. The models are free. The only cost is the compute you already own.

In my experience building production AI systems that process sensitive data at scale, the architecture that wins is the one that minimizes data movement. For security tooling, that means local inference, local storage, and zero external dependencies.

Build local. Analyze local. Keep your security data where it belongs — on your network.

About the Author

Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, focused on semantic indexing and retrieval-augmented generation (RAG). He maintains 116+ open-source repositories, including a suite of security AI tools powered by local LLMs. His work explores the intersection of AI, privacy, and practical security tooling.

GitHub: @kennedyraju55
Dev.to: nrk_raju
LinkedIn: nrk-raju-guthikonda

Your Financial Data Should Never Leave Your Machine — Here's How I Built 5 AI Tools That Prove It

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:18:22 +0000

Every day, millions of people paste their bank statements into ChatGPT. They upload invoices to cloud AI services. They feed their tax documents into web apps that promise "AI-powered analysis."

And every single one of them is handing their most sensitive data — income, spending habits, debt, investments — to a third-party server they don't control.

I'm a Senior Software Engineer at Microsoft, working on Copilot Search Infrastructure (semantic indexing, RAG pipelines). I build AI systems for a living. And I'm here to tell you: there is absolutely no reason your financial data needs to leave your machine to get intelligent analysis.

Over the past year, I've built 116+ open-source projects — and a significant chunk of them are financial AI tools that run entirely on your local hardware. No API keys. No cloud calls. No data exfiltration. Just you, your machine, and a local LLM.

Let me show you how.

The Privacy Problem Nobody Talks About

When you use a cloud-based AI service to analyze your finances, here's what actually happens:

Your data travels across the internet — bank statements, transaction histories, salary information
It lands on someone else's servers — often in a jurisdiction you didn't choose
It may be used for training — many services reserve the right to use your inputs
It persists in logs — even "deleted" data can live in backups, caches, and audit trails
It's one breach away from exposure — and financial data is the #1 target for attackers

This isn't hypothetical. As someone who works on enterprise-grade AI infrastructure at Microsoft, I've seen firsthand how seriously large organizations take data residency and privacy. The question is: why don't individuals demand the same protections?

The answer used to be "because local AI wasn't good enough." That's no longer true.

The Local LLM Stack: Ollama + Gemma

The foundation of every financial tool I've built is dead simple:

Ollama — A local LLM runtime that makes running models as easy as ollama pull gemma3
Gemma — Google's open-weight model family, optimized for efficiency
Python — Because the ecosystem for data processing is unmatched
FastAPI — For REST APIs when you want programmatic access
Streamlit — For web UIs that non-technical users can actually use

Here's what the core integration looks like:

import requests
import json

class LocalFinancialLLM:
    """Interface to local Ollama instance for financial analysis."""

    def __init__(self, model: str = "gemma3", base_url: str = "http://localhost:11434"):
        self.model = model
        self.base_url = base_url

    def analyze(self, prompt: str, context: str = "") -> str:
        """Send a financial analysis prompt to the local LLM."""
        full_prompt = f"""You are a financial analyst assistant.
Analyze the following financial data and provide actionable insights.

Context: {context}

Data/Question: {prompt}

Provide a structured analysis with key findings and recommendations."""

        response = requests.post(
            f"{self.base_url}/api/generate",
            json={
                "model": self.model,
                "prompt": full_prompt,
                "stream": False,
                "options": {
                    "temperature": 0.3,  # Low temp for financial accuracy
                    "num_predict": 2048,
                }
            }
        )
        return response.json()["response"]


# Usage — everything stays on your machine
llm = LocalFinancialLLM()
result = llm.analyze(
    prompt="Categorize these transactions and identify unusual spending",
    context="Monthly household expenses for March 2024"
)
print(result)

That's it. No API key. No cloud endpoint. No terms of service that let a corporation train on your bank statements. The model runs on your hardware, processes your data in your RAM, and the results never touch a network interface.

Five Tools, Zero Cloud Dependencies

Let me walk you through the financial AI tools I've built and open-sourced. Every single one follows the same architecture: local LLM, local data, local results.

1. Household Budget Analyzer

Repo: kennedyraju55/household-budget-analyzer

This is the tool I use for my own family's finances. Feed it a CSV of transactions and it gives you:

AI-powered spending analysis with savings recommendations
Auto-categorization — maps "Whole Foods" to Groceries, "AT&T" to Utilities
Budget vs. actual comparison — shows exactly where you're over or under
Recurring expense detection — catches hidden subscriptions
Savings goal tracking with estimated completion dates
Monthly trend analysis — visualizes spending patterns over time

from budget_analyzer.core import (
    load_expenses, compute_category_breakdown,
    detect_recurring, SavingsGoal
)

# Load your private financial data — stays 100% local
expenses = load_expenses("data/expenses.csv")
categories = compute_category_breakdown(expenses)

# Detect subscriptions you forgot about
recurring = detect_recurring(expenses)
for item in recurring:
    print(f"📌 {item['description']}: ${item['avg_amount']:.2f}/month")

# Track savings with AI-powered projections
goal = SavingsGoal(
    name="Emergency Fund",
    target_amount=10000,
    current_amount=3500,
    monthly_contribution=500
)
print(f"🎯 {goal.track_progress()}")
print(f"📅 Estimated completion: {goal.estimate_completion()}")

2. Financial Report Generator

Repo: kennedyraju55/financial-report-generator

Generates professional financial reports from raw data — the kind of output you'd expect from an analyst, but produced entirely on your laptop. Income statements, expense breakdowns, cash flow analysis, and forward-looking projections. All powered by Gemma running locally through Ollama.

3. Invoice Extractor

Repo: kennedyraju55/invoice-extractor

Drop in an invoice (text, PDF content, or structured data) and the local LLM extracts vendor name, amounts, line items, dates, and tax information into clean structured JSON. Perfect for small business owners who process dozens of invoices monthly and don't want their vendor relationships and pricing sitting on someone else's server.

4. Sentiment Analysis Dashboard

Repo: kennedyraju55/sentiment-analysis-dashboard

While not strictly a "finance" tool, this is invaluable for market analysis. Feed it earnings call transcripts, financial news, or analyst reports and get sentiment scoring with explanations. I built this with a Streamlit dashboard so you can visualize sentiment trends over time — useful for anyone doing their own investment research.

5. Stock Report Generator

Repo: kennedyraju55/stock-report-generator

AI-powered technical analysis and risk assessment for stocks. It generates structured reports covering price analysis, risk factors, and market context. Again — your investment research stays private. No cloud service knows which stocks you're analyzing or what positions you're considering.

The Architecture Pattern

Every tool follows the same proven architecture:

┌──────────────────────────────────────────────────┐
│                   Your Machine                    │
│                                                   │
│  ┌─────────────┐    ┌──────────────────────────┐ │
│  │  Streamlit   │    │     FastAPI REST API     │ │
│  │  Web UI      │    │     (localhost:8000)     │ │
│  │  (:8501)     │    │                          │ │
│  └──────┬───────┘    └────────────┬─────────────┘ │
│         │                         │               │
│         ▼                         ▼               │
│  ┌────────────────────────────────────────────┐   │
│  │          Core Analysis Engine               │   │
│  │   (Python: pandas, data processing)         │   │
│  └─────────────────────┬──────────────────────┘   │
│                        │                          │
│                        ▼                          │
│  ┌────────────────────────────────────────────┐   │
│  │         Ollama (localhost:11434)            │   │
│  │         Running Gemma 3/4 Model            │   │
│  │         ~4-8GB RAM                         │   │
│  └────────────────────────────────────────────┘   │
│                                                   │
│  ┌────────────────────────────────────────────┐   │
│  │    Local Storage (CSV/JSON)                │   │
│  │    Your data. Your machine. Your rules.    │   │
│  └────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────┘

        ❌ NOTHING crosses this boundary ❌

Key design decisions:

No outbound network calls — the entire stack runs on localhost
Config-driven — YAML files control model selection, temperature, categories
Dual interface — CLI for power users, Streamlit for everyone else
Docker-ready — docker compose up and you're running
Tested — pytest suites with 80%+ coverage on core logic

What You Need to Get Started

The hardware requirements are surprisingly modest:

Component	Minimum	Recommended
RAM	8 GB	16 GB
Storage	10 GB (for models)	20 GB
CPU	Any modern x86/ARM	Apple Silicon / GPU
GPU	Not required	NVIDIA for speed
OS	Linux, macOS, Windows	Any

Getting started takes five minutes:

# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model
ollama pull gemma3

# 3. Clone any of the tools
git clone https://github.com/kennedyraju55/household-budget-analyzer.git
cd household-budget-analyzer

# 4. Set up Python environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 5. Run it
python -m budget_analyzer.cli analyze --file expenses.csv

Why This Matters Beyond Privacy

Privacy is the headline, but it's not the only benefit:

Cost: $0 forever. No monthly API fees. No per-token charges. No surprise bills.
Speed: No network latency. Results come back in seconds, not after a round-trip to Virginia.
Reliability: No outages. Your tools work on an airplane, in a cabin with no WiFi, during a cloud provider's bad day.
Control: Swap models freely. Gemma too small? Try Llama 3. Want something lighter? Use Gemma 2B. No vendor lock-in.
Compliance: GDPR/HIPAA-friendly by default. If data never leaves your machine, there's nothing to regulate.

The Bigger Picture

I've now published 116+ open-source repositories, spanning healthcare AI, education tools, developer utilities, creative AI, and yes — financial tools. They all share the same conviction: powerful AI doesn't require surrendering your data.

Working on Copilot Search Infrastructure at Microsoft has given me a deep appreciation for what's possible with semantic indexing and RAG at scale. But it's also shown me that the most impactful AI isn't always the biggest model or the most expensive infrastructure. Sometimes it's a 4-billion-parameter model running on your laptop, analyzing your family's budget without telling anyone about it.

The tools are open source. The models are open weight. The only thing standing between you and private financial AI is a git clone.

Get Started

All projects are MIT-licensed and ready to use:

🏠 household-budget-analyzer — Budget tracking & spending analysis
📊 financial-report-generator — Professional report generation
📄 invoice-extractor — Structured data extraction from invoices
📈 sentiment-analysis-dashboard — Market sentiment scoring
📉 stock-report-generator — Technical analysis & risk assessment

Star them, fork them, make them better. And stop sending your bank statements to the cloud.

Published by **Nrk Raju Guthikonda* — Senior Software Engineer at Microsoft (Copilot Search Infrastructure). Builder of 116+ open-source AI tools. Passionate about private, local-first AI.*

Find me on GitHub · dev.to · LinkedIn

Semantic Search at Scale: What I Learned Building RAG Infrastructure at Microsoft Copilot

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:13:16 +0000

I work on Microsoft Copilot's Search Infrastructure team, where I focus on semantic indexing and RAG (Retrieval-Augmented Generation). The challenges of building search at scale are fundamentally different from what you encounter in tutorials. Here's what building production RAG taught me — and how I applied those lessons to my open-source projects.

The Tutorial vs. Production Gap

Most RAG tutorials show this:

# Tutorial RAG
chunks = split_document(document)
embeddings = embed(chunks)
store_in_vector_db(embeddings)

# At query time
query_embedding = embed(query)
results = vector_db.search(query_embedding, top_k=5)
answer = llm.generate(query + results)

This works for a demo. It fails spectacularly in production because:

Chunking strategy matters enormously — naive splitting breaks mid-sentence, mid-paragraph, mid-concept
Embedding quality varies by domain — a model trained on web text performs poorly on legal contracts or medical records
Top-k retrieval isn't enough — you need re-ranking, filtering, and relevance scoring
Context window management — stuffing 5 chunks into a prompt wastes tokens on irrelevant content
Freshness — documents update, and your index needs to stay current without full re-indexing

Lesson 1: Chunking Is an Art, Not a Split

The biggest mistake in RAG is treating chunking as text.split(max_length). Good chunking preserves:

Semantic boundaries — paragraphs, sections, logical units
Context — each chunk should be understandable in isolation
Overlap — some repetition between chunks prevents information loss at boundaries
Metadata — source document, section header, page number

class SemanticChunker:
    def chunk(self, document: str, max_tokens: int = 512) -> list:
        sections = self._split_by_headers(document)
        chunks = []
        for section in sections:
            if self._token_count(section) <= max_tokens:
                chunks.append(section)
            else:
                paragraphs = section.split('\n\n')
                chunks.extend(self._merge_paragraphs(paragraphs, max_tokens))
        return chunks

    def _merge_paragraphs(self, paragraphs, max_tokens):
        merged = []
        current = ""
        for para in paragraphs:
            if self._token_count(current + para) <= max_tokens:
                current += "\n\n" + para
            else:
                if current:
                    merged.append(current.strip())
                current = para
        if current:
            merged.append(current.strip())
        return merged

In my experience building production systems, domain-specific chunking strategies consistently outperform generic ones on retrieval relevance.

Lesson 2: Re-ranking Changes Everything

Vector similarity search returns "similar" results. Similar isn't the same as "relevant." A re-ranker bridges this gap:

def search_with_rerank(self, query: str, top_k: int = 5) -> list:
    # Phase 1: Broad retrieval (cast a wide net)
    candidates = self.vector_db.search(query, top_k=top_k * 3)

    # Phase 2: Re-rank with cross-encoder
    scored = []
    for candidate in candidates:
        score = self.reranker.score(query, candidate.text)
        scored.append((candidate, score))

    # Phase 3: Return top-k after re-ranking
    scored.sort(key=lambda x: x[1], reverse=True)
    return [item[0] for item in scored[:top_k]]

Retrieve 3x what you need, re-rank, then take the top results. This consistently improves answer quality by 15-25% over pure vector search.

Lesson 3: Hybrid Search Beats Pure Semantic

Pure embedding-based search misses exact matches. If a user searches for "error code E4012", semantic search might return results about "error handling" instead of the specific error code.

The solution is hybrid search: combine semantic similarity with keyword/BM25 matching:

def hybrid_search(self, query: str, alpha: float = 0.7) -> list:
    semantic_results = self.vector_search(query)
    keyword_results = self.bm25_search(query)

    # Reciprocal Rank Fusion
    combined = {}
    for rank, result in enumerate(semantic_results):
        combined[result.id] = combined.get(result.id, 0) + alpha / (rank + 60)
    for rank, result in enumerate(keyword_results):
        combined[result.id] = combined.get(result.id, 0) + (1 - alpha) / (rank + 60)

    return sorted(combined.items(), key=lambda x: x[1], reverse=True)

In production RAG systems, hybrid search with Reciprocal Rank Fusion is a proven approach that consistently delivers better results.

Lesson 4: Evaluation Is Non-Negotiable

You can't improve what you can't measure. Every RAG system needs:

Retrieval metrics: Recall@K, MRR, NDCG
Generation metrics: Faithfulness, relevance, completeness
End-to-end metrics: User satisfaction, task completion rate

def evaluate_retrieval(self, test_set: list) -> dict:
    recall_at_5 = []
    mrr = []

    for query, expected_docs in test_set:
        results = self.search(query, top_k=5)
        result_ids = [r.id for r in results]

        # Recall@5
        hits = len(set(result_ids) & set(expected_docs))
        recall_at_5.append(hits / len(expected_docs))

        # MRR
        for rank, rid in enumerate(result_ids):
            if rid in expected_docs:
                mrr.append(1.0 / (rank + 1))
                break
        else:
            mrr.append(0.0)

    return {
        "recall@5": sum(recall_at_5) / len(recall_at_5),
        "mrr": sum(mrr) / len(mrr)
    }

Applying These Lessons to Open Source

I've applied lessons learned from working on production-scale search to my open-source projects:

pdf-chat-assistant — RAG over PDF documents with semantic chunking and re-ranking
personal-knowledge-base — Local RAG over your personal documents
study-buddy-bot — RAG over textbook content for educational Q&A

The architecture is the same. The scale is different. But the patterns transfer perfectly.

Key Takeaways

Don't skip chunking strategy — it's the most impactful optimization
Always re-rank — the cost is minimal, the quality improvement is significant
Use hybrid search — semantic + keyword catches what pure semantic misses
Measure everything — build evaluation into your pipeline from day one
Start local — you can build and test great RAG systems on a single machine

The full collection of RAG-powered tools is on GitHub. The patterns are open source. Build better search.

Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, where he builds semantic indexing and RAG systems at scale. He maintains 116+ open-source repositories. Read more on dev.to.

The Developer's Guide to Running LLMs Locally: Ollama, Gemma 4, and Why Your Side Projects Don't Need an API Key

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:09:13 +0000

Every tutorial about building with LLMs starts the same way: "First, get your OpenAI API key." But what if I told you that you can build production-quality AI applications without ever making a cloud API call?

I've built over 90 applications using local LLMs — no API keys, no cloud costs, no rate limits. Here's a practical guide to getting started with Ollama and Gemma 4 for your own projects.

Why Local LLMs?

Before diving into the how, let's talk about why:

1. Zero Cost Per Request

Cloud APIs charge per token. A moderate application making 1,000 requests/day costs $30-100/month. Scale to production and you're looking at thousands per month. Local inference costs electricity — pennies per hour.

2. No Rate Limits

I've hit OpenAI rate limits at 3 AM on a Sunday during a hackathon. With local models, you can generate as fast as your hardware allows, 24/7.

3. Privacy by Default

No data leaves your machine. This isn't just nice-to-have — it's essential for healthcare (HIPAA), legal (attorney-client privilege), finance (PCI), and education (FERPA) applications.

4. Offline Capability

Once the model is downloaded, you need zero internet. Build on a plane. Demo without WiFi. Deploy in air-gapped environments.

5. Reproducibility

Cloud models change without notice. GPT-4 in January behaves differently than GPT-4 in June. Local models are frozen — same model, same behavior, always.

Getting Started: 5 Minutes to Your First Local LLM

Step 1: Install Ollama

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

Step 2: Pull Gemma 4

ollama pull gemma4

This downloads the model (~5GB). One-time cost, then it's on your machine forever.

Step 3: Test It

ollama run gemma4 "Explain quantum computing in one paragraph"

That's it. You now have a local LLM running on your machine.

Building Applications with Python + Ollama

Here's a minimal Python application:

import ollama

def ask(question: str) -> str:
    response = ollama.generate(
        model="gemma4",
        prompt=question,
        options={"temperature": 0.3}
    )
    return response["response"]

# That's literally it
print(ask("What are the SOLID principles in software engineering?"))

Adding Structure: The Pattern I Use in 90+ Projects

class LocalLLMApp:
    def __init__(self, model: str = "gemma4"):
        self.client = ollama.Client()
        self.model = model

    def generate(self, prompt: str, temperature: float = 0.3, 
                 system: str = None) -> str:
        messages = []
        if system:
            messages.append({"role": "system", "content": system})
        messages.append({"role": "user", "content": prompt})

        response = self.client.chat(
            model=self.model,
            messages=messages,
            options={"temperature": temperature}
        )
        return response["message"]["content"]

This base class pattern is the foundation of every application I've built. Domain-specific logic goes in subclasses — the LLM integration stays clean and swappable.

Adding a Web Interface: Streamlit

import streamlit as st

app = LocalLLMApp()

st.title("My Local AI Tool")
user_input = st.text_area("Enter your text:")

if st.button("Analyze"):
    with st.spinner("Thinking..."):
        result = app.generate(user_input)
    st.write(result)

Three imports. Ten lines. A full web interface for your local AI tool.

Adding an API: FastAPI

from fastapi import FastAPI
from pydantic import BaseModel

api = FastAPI()
app = LocalLLMApp()

class Query(BaseModel):
    text: str
    temperature: float = 0.3

@api.post("/analyze")
async def analyze(query: Query):
    result = app.generate(query.text, temperature=query.temperature)
    return {"result": result}

Now you have a REST API that any frontend, mobile app, or service can call — all running locally.

Docker: One-Command Deployment

Every project I build ships with this docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  app:
    build: .
    ports:
      - "8501:8501"
      - "8000:8000"
    depends_on:
      - ollama
    environment:
      - OLLAMA_HOST=http://ollama:11434

volumes:
  ollama-data:

docker compose up — that's the entire deployment story. Works on any machine with Docker and a GPU.

Performance: What to Expect

On consumer hardware (RTX 3080, 16GB RAM):

Simple Q&A: 0.5-1 second
Paragraph generation: 2-5 seconds
Document analysis (2-3 pages): 5-15 seconds
Long-form generation (1000+ words): 15-30 seconds

These are practical, usable response times for interactive applications.

When to Use Cloud vs. Local

Use Case	Local	Cloud
Prototyping	✅ Zero cost	❌ Token costs add up
Sensitive data	✅ Privacy by default	❌ Requires BAA/DPA
Production (small scale)	✅ Fixed hardware cost	✅ Easy to scale
Production (large scale)	❌ Hardware limits	✅ Elastic scaling
Offline/air-gapped	✅ Works anywhere	❌ Requires internet
Cutting-edge capability	❌ Smaller models	✅ Latest models

My rule: start local, move to cloud only when you've proven the concept and need scale that local hardware can't handle.

90+ Projects and Counting

I've applied this pattern across:

Healthcare: Patient intake, lab results, EHR de-identification
Legal: Contract analysis, brief generation, compliance checking
Education: Study bots, exam generators, flashcard creators
Creative: Story generators, poetry engines, mood journals
Developer Tools: Code review, API docs, performance profiling
Finance: Budget analyzers, financial report summarizers
Security: Vulnerability scanners, alert summarizers

Every single one follows the same pattern: Ollama + Gemma 4 + Python + FastAPI + Streamlit + Docker.

The code is open source: github.com/kennedyraju55

Start building locally. Your AI projects don't need an API key.

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team. He maintains 116+ original open-source repositories built with local LLMs. Read more on dev.to.*aipythontutorialdocker

Creative AI Without the Cloud: Building Story Generators, Poetry Engines, and More with Local LLMs

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:06:39 +0000

There's a common misconception that creative AI — story generation, poetry, songwriting, art descriptions — requires massive cloud models. GPT-4, Claude, Gemini — the bigger the better, right?

Not necessarily. I've built seven creative AI tools that run entirely on local hardware using Gemma 4 via Ollama. They generate compelling stories, poems, song lyrics, and creative content without sending a single byte to the cloud. Here's what I learned about making local LLMs creative.

Why Local Creative AI Matters

Creative writing involves personal expression. When you use a cloud-based AI to help with creative work, you're sharing:

Your creative ideas — which could be used for training data
Your writing style — which becomes part of the model's knowledge
Your intellectual property — stories, poems, and lyrics you co-create

For professional writers, this is a real concern. If your unpublished novel's plot gets fed into training data, who owns that idea?

Local LLMs eliminate this entirely. Your creative work stays on your machine.

The Creative AI Suite

1. Story Generator

The story generator creates narrative fiction from prompts, with control over genre, tone, length, and style.

class StoryGenerator:
    def __init__(self, model="gemma4"):
        self.client = ollama.Client()
        self.model = model

    def generate(self, premise: str, genre: str, length: str = "short",
                 style: str = "literary", pov: str = "third") -> str:
        prompt = f"""Write a {length} {genre} story in {pov} person with a {style} style.

Premise: {premise}

Requirements:
- Strong opening hook
- Vivid sensory details
- Character development
- Satisfying resolution
- Show, don't tell
- Natural dialogue (if applicable)"""

        response = self.client.generate(
            model=self.model,
            prompt=prompt,
            options={"temperature": 0.8, "top_p": 0.9}
        )
        return response["response"]

Notice the temperature: 0.8. For creative writing, we want diversity and surprise. This is the opposite of clinical or legal applications where we used 0.1-0.2. Creative AI needs to take risks.

2. Poetry Engine

The poetry engine handles multiple forms: sonnets, haiku, free verse, limericks, villanelles, and more.

POETRY_FORMS = {
    "sonnet": "14 lines, iambic pentameter, ABAB CDCD EFEF GG rhyme scheme",
    "haiku": "3 lines: 5-7-5 syllables, nature imagery",
    "free_verse": "No fixed meter or rhyme, but with rhythm and imagery",
    "limerick": "5 lines, AABBA rhyme, humorous",
    "villanelle": "19 lines, 5 tercets + 1 quatrain, two refrains",
}

def compose(self, theme: str, form: str, mood: str = "contemplative") -> str:
    form_rules = POETRY_FORMS.get(form, "free form poetry")
    prompt = f"""Compose a {form} poem about {theme} with a {mood} mood.

Form requirements: {form_rules}

Write with:
- Vivid imagery and metaphor
- Emotional resonance
- Precise word choice
- Natural rhythm even within formal constraints"""

3. Song Lyric Writer

Generates lyrics with verse-chorus structure, including chord suggestions.

4. Family Story Creator

A unique tool that generates personalized stories for children using family members as characters. Parents input names and traits, and the AI creates bedtime stories featuring their actual family.

5. Creative Writing Coach

Analyzes drafts and provides constructive feedback on:

Pacing and structure
Character voice consistency
Show vs. tell balance
Dialogue naturalness
Opening hook strength

The Temperature Spectrum

The most important lesson from building both clinical and creative AI tools is understanding the temperature parameter:

Application	Temperature	Why
Clinical summarization	0.1	Accuracy over creativity
Legal analysis	0.2	Reasoning with minimal hallucination
Code generation	0.3	Correct syntax, some flexibility
Educational content	0.5	Balanced: accurate but engaging
Business writing	0.6	Professional but not robotic
Creative fiction	0.8	Surprising, expressive
Poetry/experimental	0.9	Maximum creativity

This spectrum emerged from building 90+ tools across every domain. It's not in any textbook — it's practical knowledge from thousands of generations across different use cases.

Running on Consumer Hardware

All seven creative tools run on a single machine with:

GPU: RTX 3080 or equivalent (8GB+ VRAM)
RAM: 16GB system RAM
Storage: 10GB for the model

Generation times:

Short story (500 words): 5-10 seconds
Poem: 2-4 seconds
Song lyrics: 4-8 seconds
Story feedback: 8-15 seconds

Fast enough for interactive creative sessions where you generate, read, regenerate, and iterate.

Open Source Creative Tools

All tools are available on GitHub:

family-story-creator — Personalized family stories
mood-journal-bot — AI-powered mood journaling
standup-generator — Creative standup comedy bits
diary-journal-organizer — Smart journal organization

Creative AI doesn't need to be a cloud service. With local LLMs, your creative work stays private, runs fast, and costs nothing per generation.

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team. He maintains 116+ original open-source repositories spanning healthcare, legal, education, creative AI, and developer tools. Find his work on GitHub and dev.to.*aicreativepythonwriting

AI Tutoring That Works Offline: Building Education Tools That Don't Need the Internet

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:03:06 +0000

The digital divide in education isn't just about having a device — it's about having reliable internet. According to the FCC, over 14 million students in the US lack adequate home internet access. Cloud-based AI tutoring tools are useless to them.

I built a suite of education AI tools that run entirely offline using local LLMs. No internet required after initial setup. No student data leaving the device. No subscription costs per student.

Here's how local AI can democratize education technology.

The Connectivity Problem

Most "AI in education" products assume always-on internet:

ChatGPT needs cloud access for every interaction
Khan Academy's AI tutor requires constant connectivity
Google's learning tools depend on Google Cloud

This creates a two-tier education system: students with fast internet get AI-powered learning, and everyone else gets left behind. Rural schools, developing countries, and low-income households — the communities that need educational technology most — can't use it.

The Solution: Education AI That Runs Locally

My education tools use Gemma 4 via Ollama to provide:

1. Study Buddy Bot

An interactive study companion that helps students work through problems step-by-step.

class StudyBuddy:
    def __init__(self, subject: str, level: str):
        self.subject = subject
        self.level = level  # elementary, middle, high, college
        self.conversation_history = []

    def help_with_problem(self, problem: str) -> str:
        prompt = f"""You are a patient {self.level}-level {self.subject} tutor.

A student asks for help with: {problem}

Rules:
1. Don't give the answer directly
2. Guide them through the reasoning step by step
3. Ask questions to check understanding
4. Use analogies appropriate for {self.level} level
5. Encourage when they make progress
6. Correct misconceptions gently

Provide the next step of guidance."""

        return self._generate(prompt)

The key design choice: never give the answer directly. The Socratic method works better for learning, and it's also safer — if the LLM makes a mistake in the final answer, the student learns the wrong thing. But if the LLM asks guiding questions, the student develops their own understanding.

2. Exam Generator

Creates practice exams calibrated to curriculum standards.

def generate_exam(self, topic: str, difficulty: str, num_questions: int, 
                  question_types: list = ["multiple_choice", "short_answer", "essay"]) -> dict:
    prompt = f"""Generate a {difficulty} difficulty exam on {topic}.

Requirements:
- {num_questions} questions total
- Mix of types: {', '.join(question_types)}
- Multiple choice: 4 options each, one correct
- Include an answer key with explanations
- Align with standard curriculum for this topic
- Vary Bloom's taxonomy levels (remember, understand, apply, analyze)

Output as structured JSON."""

Teachers can generate unlimited practice exams without internet. Each generation is unique, reducing cheating on take-home practice.

3. Reading List Manager

Analyzes reading materials and suggests study paths.

Extracts key concepts from textbook chapters
Identifies prerequisite knowledge gaps
Suggests reading order for optimal learning
Generates chapter summaries and study guides

4. Flashcard Generator

Transforms any text into spaced-repetition flashcards.

Extracts key terms and definitions
Creates question-answer pairs from content
Generates "explain this concept" cards for deeper understanding
Exports to Anki-compatible format

Why This Matters

The total cost to deploy these tools is a one-time hardware purchase — a laptop with a decent GPU. After downloading the model once (with internet), everything runs offline indefinitely.

Compare this to per-student SaaS subscriptions that cost schools thousands annually, require constant internet, and send student data to cloud servers (a FERPA concern).

Deployment for Schools

Every tool ships with Docker for easy deployment:

# One command to start the entire education AI suite
docker compose up

A school IT administrator can set up a single server that runs all the tools. Students connect via the local network — no internet needed. Student interactions stay on the school's own hardware.

For individual students, the tools run on any laptop with 8GB+ RAM. Not gaming-PC specs — just a reasonable modern laptop.

What's Next

I'm working on:

Curriculum alignment — mapping generated content to Common Core and state standards
Multi-language support — Gemma 4 supports multiple languages, enabling tutoring in students' native languages
Teacher dashboards — aggregated (anonymized) analytics showing class-wide concept gaps
Accessibility — screen reader support and simplified interfaces for students with disabilities

The tools are open source:

study-buddy-bot — Interactive AI study companion
exam-generator — Practice exam generation
reading-list-manager — Smart reading path suggestions
flashcard-generator — Automated flashcard creation

Education AI should work for every student, not just those with fast internet and school budgets for SaaS subscriptions. Local LLMs make that possible today.

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team. He builds privacy-first AI tools across healthcare, legal, education, and enterprise domains. Explore his 116+ open-source repositories on GitHub and read more on dev.to.*educationaipythonopensource

From Side Projects to 116 Repositories: How I Built an Open-Source AI Portfolio While Working Full-Time at Microsoft

Nrk Raju Guthikonda — Sun, 12 Apr 2026 23:01:36 +0000

Two years ago, I had a handful of GitHub repositories — mostly experimental scripts and weekend hacks. Today, I maintain 116 original repositories spanning healthcare AI, legal tech, developer tools, creative AI, education, finance, and security.

Every single one is original work. Zero forks. All built with a consistent philosophy: AI should run locally, respect privacy, and solve real problems.

Here's what I learned building this portfolio while working full-time as a Senior Software Engineer on Microsoft's Copilot Search Infrastructure team.

The 90-Local-LLM Rule

Early on, I made a decision that shaped everything: every AI project would run locally. No cloud API keys. No data transmission. No per-token costs.

This wasn't just a technical preference — it was a product thesis. I believe the future of AI isn't centralized cloud APIs but distributed local inference. And I wanted to prove it was practical by building 90+ working applications across every domain I could think of.

The stack is consistent across all projects:

Gemma 4 (or earlier Gemma models) via Ollama for inference
Python for core logic
FastAPI for API layers
Streamlit for user interfaces
Docker for deployment

This consistency means each project builds on patterns from previous ones. The tenth healthcare tool took a fraction of the time of the first because the architecture was battle-tested.

Picking Domains That Matter

I didn't build 116 "todo app with AI" variations. Each project targets a real problem in a specific domain:

Healthcare (15+ repos)

Patient intake summarizers that keep PHI on-premise
Lab result interpreters with clinical context
EHR de-identification tools
Medical document assistants
Mental health check-in tools

The healthcare tools are built around a single principle: no patient data should ever leave the hospital's network. Every one runs entirely offline after initial model download.

Legal Tech (8+ repos)

Contract clause analyzers
Legal brief generators
Compliance checkers
Court case summarizers

Legal AI has the same confidentiality imperative as healthcare — attorney-client privilege doesn't survive a round trip to a cloud API.

Developer Tools (20+ repos)

Code review assistants
API documentation generators
Git analytics dashboards
Performance profiling tools

These are tools I actually use in my day job. Building them made me a better engineer, and open-sourcing them helped others.

Education, Finance, Security, Creative AI (50+ repos)

Exam generators and tutoring bots
Financial report analyzers
Security audit tools
Story generators and poetry engines

Each domain taught me something about how LLMs interact with domain-specific knowledge. Medical terminology behaves differently than legal jargon, which behaves differently than financial reporting language. The prompting strategies that work for clinical summarization fail for creative writing.

The Architecture Pattern

After 116 repos, I've converged on a pattern that works:

project/
├── src/
│   ├── core/          # Domain logic (no LLM dependency)
│   ├── llm/           # LLM integration layer
│   ├── api/           # FastAPI endpoints
│   └── ui/            # Streamlit interface
├── tests/
├── docker-compose.yml # One-command deployment
├── README.md          # Problem, solution, architecture, demo
└── .env.example       # Configuration template

Key principles:

Separate domain logic from LLM integration — the core business logic should work with any model, or even without one
Always provide both API and UI — API for integration, UI for demos and non-technical users
Docker-first deployment — docker compose up should be the only command needed
Comprehensive README — every project explains the problem it solves, not just how to run it

Time Management: Building While Working Full-Time

The most common question I get: "How do you build this much while working full-time?"

The honest answer:

Reuse patterns aggressively — that project template above means I can scaffold a new project in 20 minutes
Build in domains you know — working on Copilot Search taught me RAG patterns that directly informed my retrieval-augmented projects
Small, focused projects — each repo solves one problem well. A contract analyzer doesn't try to also manage cases
Weekend sprints — most projects start as Saturday afternoon prototypes. If the prototype works, it gets a full README and Docker setup the next day
Automate everything else — I have scripts for repo creation, README generation, and deployment

What This Portfolio Has Done for My Career

Building 116 original repositories has:

Deepened my expertise — you don't truly understand RAG until you've built it for healthcare, legal, and education domains
Created a public body of work — every repo is a verifiable, runnable demonstration of skill
Opened conversations — colleagues and recruiters reference specific projects
Contributed to open source — over 50 projects have README-driven documentation that helps others learn
Built credibility in AI/ML — a portfolio this size, with this consistency, demonstrates sustained commitment

Advice for Building Your Own Portfolio

If you're considering building a similar open-source portfolio:

Pick a consistent stack — don't learn a new framework for each project. Master one stack and push it to its limits
Solve real problems — "GPT wrapper" projects don't demonstrate skill. Privacy-first healthcare AI demonstrates both technical ability and domain understanding
Write the README first — if you can't explain the problem and solution clearly, the project isn't ready
Ship Docker — if someone can't run your project with a single command, they won't try it
Be original — forking and modifying existing projects teaches less than building from scratch
Stay consistent — 116 repos didn't happen overnight. Commit to building something new every week

The full portfolio is available at github.com/kennedyraju55 and showcased at kennedyraju55.github.io.

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, specializing in semantic indexing and RAG systems. He maintains 116+ original open-source repositories. Read more on dev.to.*opensourceaicareerprogramming

Contract Analysis with Local LLMs: Why Law Firms Should Stop Sending Documents to the Cloud

Nrk Raju Guthikonda — Sun, 12 Apr 2026 22:58:00 +0000

Legal documents are among the most sensitive files in any organization. Yet the current wave of "AI-powered contract review" tools wants you to upload those documents to cloud APIs — exposing client confidentiality, attorney-client privilege, and trade secrets to third-party servers.

I built an alternative: a contract clause analyzer that runs entirely on your machine using Gemma 4 via Ollama. Zero cloud transmission. Complete confidentiality. Here's how it works and why it matters.

The Confidentiality Problem

When a law firm uploads a contract to a cloud-based AI tool, several things happen:

Attorney-client privilege may be waived — transmitting privileged documents to a third party without proper safeguards can constitute a waiver
Client data leaves your control — even with encryption, the cloud provider processes the text in plaintext during inference
Regulatory exposure increases — GDPR, CCPA, and industry regulations impose strict requirements on data processing locations
Competitive intelligence leaks — M&A contracts, employment agreements, and IP licenses contain strategic information

The American Bar Association's Model Rule 1.6 requires lawyers to make "reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to, information relating to the representation of a client." Sending contracts to a cloud LLM is a gray area that many ethics committees are actively scrutinizing.

Architecture: Local-First Contract Analysis

My contract-clause-analyzer uses a three-stage pipeline:

┌─────────────────────────────────────────┐
│         Document Input Layer            │
│  (PDF, DOCX, TXT parsing + OCR)        │
├─────────────────────────────────────────┤
│         Clause Extraction Engine        │
│  (Section splitting, clause typing,     │
│   reference resolution)                 │
├─────────────────────────────────────────┤
│         LLM Analysis Layer              │
│  (Gemma 4 via Ollama — local only)     │
│  Risk scoring, term comparison,         │
│  plain-English summaries                │
└─────────────────────────────────────────┘
         ↕ Everything stays on localhost

Stage 1: Document Parsing

Contracts come in every format. The parser handles:

PDF — both text-based and scanned (with Tesseract OCR fallback)
DOCX — preserving section structure and numbering
Plain text — for already-extracted content

The key challenge is preserving document structure. A clause that says "Subject to Section 4.2(a)" needs to be linked to that section. The parser builds a section tree that maintains these cross-references.

Stage 2: Clause Extraction

Not every paragraph in a contract is a "clause" worth analyzing. The extraction engine identifies:

Operative clauses — obligations, rights, conditions
Boilerplate — standard terms that still matter (governing law, dispute resolution, force majeure)
Definitions — terms that affect interpretation of other clauses
Schedules and exhibits — referenced attachments

Each clause is classified by type:

CLAUSE_TYPES = {
    "indemnification": ["indemnif", "hold harmless", "defend and indemnify"],
    "limitation_of_liability": ["limitation of liability", "aggregate liability", "cap on damages"],
    "termination": ["terminat", "expiration", "cancellation rights"],
    "confidentiality": ["confidential", "non-disclosure", "proprietary information"],
    "ip_assignment": ["intellectual property", "work product", "assignment of rights"],
    "non_compete": ["non-compete", "non-solicitation", "restrictive covenant"],
    "payment_terms": ["payment", "invoice", "net 30", "compensation"],
    "warranty": ["warrant", "represent", "guarantee"],
    "force_majeure": ["force majeure", "act of god", "beyond reasonable control"],
    "governing_law": ["governing law", "jurisdiction", "venue"],
    "dispute_resolution": ["arbitration", "mediation", "dispute resolution"],
    "data_protection": ["data protection", "GDPR", "personal data", "privacy"]
}

Stage 3: LLM Analysis

This is where Gemma 4 shines. For each extracted clause, the LLM provides:

Risk Assessment:

def analyze_clause(self, clause_text: str, clause_type: str, party_role: str) -> dict:
    prompt = f"""You are a contract analysis assistant. Analyze this {clause_type} clause 
from the perspective of the {party_role} (the party you are advising).

Clause:
{clause_text}

Provide:
1. RISK_LEVEL: HIGH, MEDIUM, or LOW
2. KEY_ISSUES: List specific concerns (max 5)
3. MISSING_PROTECTIONS: What standard protections are absent
4. PLAIN_ENGLISH: Explain what this clause means in simple terms
5. NEGOTIATION_POINTS: Suggested changes to improve the party's position

Be specific. Reference exact language from the clause."""

    response = self.client.generate(
        model="gemma4",
        prompt=prompt,
        options={"temperature": 0.2}
    )
    return self._parse_analysis(response["response"])

The temperature is set to 0.2 — slightly higher than clinical applications because legal analysis benefits from some reasoning diversity, but still low enough to avoid hallucinating contract terms that don't exist.

Comparative Analysis:

The tool can also compare clauses against a library of "standard" terms:

def compare_to_standard(self, clause_text: str, clause_type: str) -> dict:
    standard = self.standard_library.get(clause_type)
    if not standard:
        return {"comparison": "No standard template available for this clause type"}

    prompt = f"""Compare this contract clause to the standard template below.

ACTUAL CLAUSE:
{clause_text}

STANDARD TEMPLATE:
{standard}

Identify:
1. DEVIATIONS: Where the actual clause differs from standard
2. FAVORABLE_TERMS: Terms that are better than standard (for our client)
3. UNFAVORABLE_TERMS: Terms that are worse than standard
4. MISSING_TERMS: Standard protections that are absent"""

This is incredibly powerful for junior associates who need to review contracts against firm templates — they get instant markup of deviations without senior partner time.

Real-World Use Cases

1. M&A Due Diligence

During an acquisition, the legal team might review hundreds of contracts. The tool can:

Batch-process all vendor agreements
Flag non-standard termination clauses (change of control triggers)
Identify IP assignment gaps
Summarize aggregate exposure from indemnification clauses

2. Employment Agreement Review

HR and legal teams can:

Compare non-compete scopes across different state jurisdictions
Flag overly broad IP assignment clauses
Ensure severance terms are consistent across employee levels
Identify clauses that may not be enforceable in specific states

3. Vendor Contract Management

Procurement teams can:

Score vendor contracts by risk level
Track SLA terms across multiple vendors
Flag auto-renewal clauses before they trigger
Ensure data protection addenda are present and adequate

Performance

On a consumer GPU (RTX 3080):

Single clause analysis: 1-3 seconds
Full contract (50 pages): 2-5 minutes
Batch processing (100 contracts): 3-4 hours unattended

These times are comparable to cloud APIs — but without the per-token costs that make batch processing expensive. A 50-page contract might cost $2-5 in cloud API tokens. Locally, after the one-time hardware investment, the marginal cost is electricity.

Why This Matters for Legal AI

The legal industry is at an inflection point with AI. Firms that adopt AI will outcompete those that don't. But adopting cloud-based AI for sensitive legal work creates risks that may outweigh the benefits.

Local LLMs offer a third path: the productivity gains of AI without the confidentiality risks of cloud processing. As models like Gemma 4 continue to improve, the quality gap between local and cloud inference will continue to shrink.

The code is open source and ready to deploy:

contract-clause-analyzer — Full contract analysis pipeline
legal-brief-generator — Generate legal brief drafts from case notes
ai-compliance-checker — Regulatory compliance analysis

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team. He builds privacy-first AI tools across healthcare, legal, and enterprise domains. Explore his 116+ open-source repositories on GitHub and read more on dev.to.*aipythonlegalprivacy

How I Built a Privacy-First Healthcare AI Agent Using MCP and Local LLMs

Nrk Raju Guthikonda — Sun, 12 Apr 2026 22:54:45 +0000

Most healthcare AI demos have a fatal flaw: they send patient data to the cloud. That's not just a bad practice — it's a regulatory minefield. HIPAA violations can cost $50,000 per incident, and "but our AI vendor said it was secure" isn't a defense.

I decided to build healthcare AI tools that solve this problem at the architecture level. No patient data ever leaves the machine. Zero cloud API calls. Complete HIPAA compliance by design, not by policy.

Here's how I built a suite of healthcare AI agents — including a patient intake summarizer, lab results interpreter, EHR de-identifier, and medical document assistant — all running locally with Gemma 4 via Ollama.

The Problem with Cloud-Based Healthcare AI

Every time a healthcare organization sends patient data to a cloud LLM API, they're creating:

A HIPAA liability — PHI (Protected Health Information) transmitted to a third party requires a Business Associate Agreement, encryption in transit and at rest, and audit trails
A single point of failure — API outages mean your clinical workflow stops
A cost that scales linearly — every patient encounter means another API call, and token costs add up fast in healthcare where documents are long
A trust problem — patients and providers increasingly ask "where does my data go?"

The solution isn't to avoid AI — it's to bring the AI to the data instead of sending the data to the AI.

Architecture: Local LLM + MCP Pattern

My architecture uses three core components:

┌─────────────────────────────────────────────┐
│           Clinical Application              │
│  (Streamlit UI / FastAPI / CLI)             │
├─────────────────────────────────────────────┤
│           MCP Server Layer                  │
│  (Tool definitions, prompt templates,       │
│   FHIR resource handlers)                  │
├─────────────────────────────────────────────┤
│           Ollama Runtime                    │
│  (Gemma 4 model, local inference,          │
│   zero network transmission)               │
└─────────────────────────────────────────────┘
         ↕ Everything stays on localhost

The Model Context Protocol (MCP) layer is what makes this modular. Instead of hardcoding LLM interactions, each healthcare capability is exposed as an MCP tool:

summarize_intake — processes patient intake forms into structured clinical summaries
interpret_lab_results — analyzes lab values against reference ranges with clinical context
deidentify_ehr — strips PHI from electronic health records while preserving clinical meaning
analyze_document — multi-agent document analysis for medical records

Why MCP?

MCP provides a standardized interface between AI models and tools. For healthcare, this means:

Interoperability — any MCP-compatible client can use the healthcare tools
Composability — chain multiple tools (e.g., de-identify → summarize → flag risks)
Testability — each tool can be tested independently with known inputs/outputs
Audit trail — every tool invocation is logged with inputs and outputs

Building the Patient Intake Summarizer

Let me walk through one tool in detail. The Patient Intake Summarizer takes unstructured intake forms and produces structured clinical summaries.

The Challenge

Patient intake forms are messy. They contain free-text descriptions mixed with medical terminology, abbreviations, and varying formats. A typical intake might read:

"52F, presenting with lower back pain x 3 weeks, worse with sitting. PMH: DM2 on metformin 500mg BID, HTN on lisinopril 10mg daily. No known allergies. Family hx: mother had MI at 62."

A clinician can parse this instantly. An LLM needs structured prompting to extract the same information reliably.

The Solution

class IntakeSummarizer:
    def __init__(self, model="gemma4"):
        self.client = ollama.Client()
        self.model = model

    def summarize(self, intake_text: str, format: str = "structured") -> dict:
        prompt = self._build_prompt(intake_text, format)
        response = self.client.generate(
            model=self.model,
            prompt=prompt,
            options={"temperature": 0.1}  # Low temp for clinical accuracy
        )
        return self._parse_response(response["response"], format)

    def _build_prompt(self, text: str, format: str) -> str:
        return f"""You are a clinical documentation assistant. 
Summarize the following patient intake form into a {format} summary.

IMPORTANT: Extract ALL of the following categories:
- Demographics (age, sex, presenting complaint)
- Medical History (conditions, surgeries, hospitalizations)  
- Current Medications (drug, dose, frequency)
- Allergies (drug, food, environmental)
- Family History (conditions, relationships)
- Social History (occupation, habits, living situation)
- Risk Factors (clinical flags requiring attention)
- Missing Information (gaps that need follow-up)

Intake Form:
{text}

Provide the summary in structured JSON format."""

The key insight is temperature 0.1. For creative writing, you want high temperature. For clinical summarization, you want the model to be as deterministic and faithful to the source text as possible. Hallucinated medical information isn't creative — it's dangerous.

Multi-Format Output

The summarizer supports three output formats:

Brief — 2-3 sentence overview for quick triage
Detailed — paragraph-form comprehensive summary
Structured — JSON with categorized fields for EHR integration

The structured format is particularly valuable because it can be directly ingested by downstream systems — no manual re-entry, no copy-paste errors.

Lab Results Interpreter

The lab interpreter is more complex because it needs reference ranges and clinical context.

REFERENCE_RANGES = {
    "glucose_fasting": {"low": 70, "high": 100, "unit": "mg/dL", "critical_low": 50, "critical_high": 400},
    "hba1c": {"low": 4.0, "high": 5.6, "unit": "%", "critical_high": 14.0},
    "creatinine": {"low": 0.7, "high": 1.3, "unit": "mg/dL", "critical_high": 10.0},
    # ... 50+ lab values
}

def interpret(self, lab_name: str, value: float, patient_context: str = "") -> dict:
    ref = REFERENCE_RANGES.get(lab_name)
    status = self._classify_value(value, ref)

    # Only call LLM for abnormal values or when context matters
    if status != "normal" or patient_context:
        interpretation = self._llm_interpret(lab_name, value, status, patient_context)
    else:
        interpretation = f"{lab_name} is within normal range."

    return {
        "lab": lab_name,
        "value": value,
        "reference_range": f"{ref['low']}-{ref['high']} {ref['unit']}",
        "status": status,
        "interpretation": interpretation
    }

Notice the optimization: we only call the LLM for abnormal values or when patient context might change the interpretation. A normal glucose in a diabetic patient means something different than in a healthy patient — that's when the LLM adds value. For straightforward normal results, a rule-based response is faster and just as accurate.

EHR De-identification

De-identification is critical for research, training, and any scenario where clinical data needs to be shared without exposing patient identity.

The tool identifies and removes 18 HIPAA identifier categories:

Names, dates, phone numbers, emails
Social Security numbers, medical record numbers
Geographic data smaller than a state
Biometric identifiers, device identifiers
URLs, IP addresses, account numbers

The LLM approach has an advantage over regex-based de-identification: it understands context. "Dr. Smith recommended the Smith protocol" — the first "Smith" is PHI, the second is a medical protocol name. A regex would remove both; the LLM preserves the clinically meaningful reference.

Docker Deployment

Every tool ships with Docker Compose for one-command deployment:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  app:
    build: .
    ports:
      - "8501:8501"  # Streamlit UI
      - "8000:8000"  # FastAPI API
    depends_on:
      - ollama
    environment:
      - OLLAMA_HOST=http://ollama:11434

docker compose up and you have a fully functional healthcare AI tool running locally. No API keys, no cloud accounts, no data leaving your network.

Results and Impact

Across the four healthcare tools, the architecture delivers:

Zero data transmission — verified with network monitoring, no outbound connections during inference
Sub-second response times — Gemma 4 on a consumer GPU generates clinical summaries in 800ms-2s
Consistent accuracy — low temperature + structured prompting produces reliable, reproducible outputs
Complete audit trail — every tool invocation logged with timestamp, input hash, and output

What's Next

I'm currently exploring:

FHIR R4 integration — mapping tool outputs to FHIR resources for EHR interoperability
A2A (Agent-to-Agent) protocol — enabling healthcare agents to collaborate (e.g., intake summarizer triggers lab interpreter which triggers risk assessment)
Federated evaluation — benchmarking accuracy across institutions without sharing data

The code is open source. If you're building healthcare AI that respects patient privacy, check out the repos:

*Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team. He has built 116+ open-source repositories, including a suite of privacy-first healthcare AI tools. Find his work on GitHub and dev.to.*healthcareaipythonsecurity

Building RAG Pipelines That Actually Work: Lessons from Microsoft Copilot

Nrk Raju Guthikonda — Sun, 12 Apr 2026 22:45:40 +0000

Most RAG tutorials show you the happy path. You chunk a handful of PDFs, toss them into a vector store, wire up an LLM, and — magic — your chatbot answers questions about your documents. Demo complete. Applause.

Here's what those tutorials don't show you: what happens when you deploy RAG at scale. When your corpus isn't 10 PDFs but 10 million documents. When your latency budget is 200 milliseconds, not "however long it takes." When a wrong answer isn't a minor inconvenience but a trust-destroying event for millions of users.

I work on Microsoft Copilot's Search Infrastructure team, where my focus is semantic indexing and retrieval-augmented generation. I've also built over 116 open-source repositories, many of which experiment with RAG patterns across healthcare, developer tools, education, and creative AI. What follows is a distillation of what I've learned — the patterns that survive contact with production, and the failure modes that tutorials conveniently skip.

What RAG Actually Is (Quick Refresher)

Retrieval-Augmented Generation is a simple idea: instead of asking an LLM to answer from memory alone, you first retrieve relevant documents, then feed them as context alongside the user's query. The basic flow:

User Query → Embed → Retrieve from Index → Augment Prompt → Generate Response

That five-step pipeline hides an enormous amount of complexity. Every arrow in that diagram is a place where things can go wrong. Let's walk through each stage and talk about what actually matters.

Chunking Strategies That Matter

Chunking is the most underrated part of the RAG pipeline. Get it wrong and nothing downstream can save you — not a better embedding model, not a smarter LLM, not a fancier retrieval algorithm. Garbage chunks in, garbage answers out.

Fixed-Size Chunking

The naive approach: split every N tokens. It's fast, deterministic, and almost always wrong. A 512-token window doesn't care that it just sliced a paragraph in half, separated a code function from its docstring, or split a table across two chunks. The resulting fragments lack semantic coherence, which means your embeddings will be noisy and your retrieval will suffer.

Semantic Chunking

A better approach respects the natural boundaries of text. Sentences, paragraphs, sections — these are the units humans write in, and they're the units that produce coherent embeddings. The key insight is that a chunk should be a self-contained unit of meaning.

Recursive and Hierarchical Chunking

For structured documents (markdown, HTML, code), recursive chunking splits along structural boundaries first — headers, then paragraphs, then sentences — falling back to smaller splits only when a section exceeds your token budget. This preserves the document's inherent hierarchy and produces chunks that actually make sense.

Overlapping Windows

Here's a pattern that pays for itself immediately: overlap between adjacent chunks. Without overlap, information that spans a chunk boundary is effectively invisible to retrieval. A query about concept X might match the end of chunk 4 and the beginning of chunk 5, but neither chunk alone scores high enough to be retrieved.

def semantic_chunk(text, max_tokens=512, overlap=50):
    sentences = split_into_sentences(text)
    chunks, current_chunk = [], []
    current_tokens = 0
    for sentence in sentences:
        tokens = count_tokens(sentence)
        if current_tokens + tokens > max_tokens and current_chunk:
            chunks.append(" ".join(current_chunk))
            # Keep overlap sentences
            overlap_sentences = current_chunk[-2:]
            current_chunk = overlap_sentences
            current_tokens = sum(count_tokens(s) for s in current_chunk)
        current_chunk.append(sentence)
        current_tokens += tokens
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    return chunks

I keep two trailing sentences as overlap. This is a deliberate choice — enough context to preserve cross-boundary meaning, but not so much that you're bloating your index with redundant content. Tune the overlap to your domain: technical documentation tends to need more overlap than conversational text.

Embedding Models — Choosing Wisely

Your embedding model is the lens through which your entire corpus is viewed. Choose poorly and retrieval becomes a game of chance.

The Landscape

OpenAI's text-embedding-ada-002 is the default choice for many teams, and it's a solid baseline — 1536 dimensions, reasonable performance across domains, easy API integration. But it's not always the right answer. Open-source models like BGE-large, E5-large-v2, and the sentence-transformers family offer competitive quality with significant advantages: no API costs at scale, lower latency (run locally or on your own GPU fleet), and the ability to fine-tune on your domain.

Domain-Specific vs. General-Purpose

If your corpus is specialized — legal documents, medical literature, codebases — a general-purpose embedding model may not capture the nuances that matter. A model fine-tuned on biomedical text will understand that "MI" means "myocardial infarction," not "Michigan." The MTEB leaderboard is your friend here: benchmark models against your actual query distribution, not generic benchmarks.

Dimensionality Tradeoffs

Higher dimensions capture more nuance but cost more in storage and search latency. At scale, the difference between 384 and 1536 dimensions is not academic — it's the difference between fitting your index in memory or needing distributed infrastructure. I've seen 768-dimensional models outperform 1536-dimensional ones on domain-specific tasks after fine-tuning. Measure, don't assume.

Asymmetric Embedding

This is the insight that separates production RAG from tutorial RAG: the query and the document should not be embedded the same way. A query like "How do I reset my password?" is semantically different from a documentation passage that contains the answer. Models like E5 handle this explicitly with query: and passage: prefixes. If your embedding model supports asymmetric encoding, use it. The retrieval quality improvement is substantial and essentially free.

Retrieval Ranking — Beyond Cosine Similarity

Cosine similarity against a dense vector index is the starting point, not the finish line. In production, you need a ranking pipeline, not a single similarity score.

Hybrid Search: Dense + Sparse

Dense retrieval (vector search) excels at semantic matching — it understands that "automobile" and "car" are related. Sparse retrieval (BM25, keyword matching) excels at exact matching — it knows that "error code 0x8007045D" is a precise string, not a semantic concept. Neither alone is sufficient. The winning combination is both.

def hybrid_search(query, index, bm25_index, k=10, alpha=0.7):
    dense_results = index.search(embed(query), k=k*2)
    sparse_results = bm25_index.search(query, k=k*2)
    # Reciprocal Rank Fusion
    scores = {}
    for rank, doc_id in enumerate(dense_results):
        scores[doc_id] = scores.get(doc_id, 0) + alpha / (rank + 60)
    for rank, doc_id in enumerate(sparse_results):
        scores[doc_id] = scores.get(doc_id, 0) + (1 - alpha) / (rank + 60)
    return sorted(scores, key=scores.get, reverse=True)[:k]

Reciprocal Rank Fusion (RRF) is elegant because it doesn't require score normalization — it works purely on rank positions. The constant 60 in the denominator is a standard dampening factor that prevents top-ranked results from dominating. The alpha parameter controls the dense-vs-sparse balance; 0.7 is a reasonable starting point, but you should tune it against your evaluation set.

Re-Ranking with Cross-Encoders

Bi-encoders (your embedding model) are fast because they encode queries and documents independently. Cross-encoders are accurate because they process the query-document pair jointly, capturing fine-grained interactions. The pattern: retrieve broadly with a bi-encoder, then re-rank the top candidates with a cross-encoder. Models like cross-encoder/ms-marco-MiniLM-L-12-v2 can re-rank 100 candidates in milliseconds.

Metadata Filtering

Not all retrieval should be purely semantic. If a user asks about "Python 3.11 features," you should filter by language and version before running vector search, not after. Pre-filtering reduces the search space and eliminates false positives that would otherwise waste context window budget.

The "Lost in the Middle" Problem

Research from Stanford showed that LLMs pay disproportionate attention to the beginning and end of their context window, often ignoring information in the middle. This has direct implications for how you order retrieved passages. Place your most relevant chunks at the beginning of the context, not in order of retrieval rank. Or better yet, interleave high and low relevance to force the model to attend evenly.

Context Window Management

You've retrieved your chunks. Now you need to fit them — along with a system prompt, the user's query, and room for the response — into a fixed token budget. This is a packing problem, and it deserves more attention than it gets.

Token Budgeting

Be explicit about your budget allocation:

TOTAL_CONTEXT = 8192  # or 128k, depends on your model
SYSTEM_PROMPT_TOKENS = 500
RESPONSE_RESERVE = 1024
USER_QUERY_TOKENS = 200  # estimate or measure

CONTEXT_BUDGET = TOTAL_CONTEXT - SYSTEM_PROMPT_TOKENS - RESPONSE_RESERVE - USER_QUERY_TOKENS
# = 6468 tokens for retrieved passages

Every token you spend on a low-relevance chunk is a token you can't spend on a high-relevance one. Rank your chunks by retrieval score and pack greedily until the budget is full.

Compression

When your top chunks exceed the budget, you have two choices: drop chunks or compress them. Compression techniques range from simple extractive summarization (keep only the most relevant sentences within each chunk) to LLM-based summarization. The tradeoff is latency vs. information density. In latency-sensitive pipelines, extractive approaches win.

Strategy Selection: Stuff vs. Map-Reduce vs. Refine

Stuff: Concatenate all retrieved chunks into a single prompt. Simple, fast, works when everything fits in the context window.
Map-Reduce: Process each chunk independently, then aggregate the results. Necessary when the total retrieved content exceeds the context window. More LLM calls, higher latency, but handles scale.
Refine: Process chunks sequentially, refining the answer with each new chunk. Produces high-quality answers but has the highest latency. Use for offline or batch workloads.

In production, I default to Stuff with aggressive filtering. If your retrieval and ranking are good, you shouldn't need more than 5–8 highly relevant chunks to answer most questions.

Common Failure Modes (And How to Debug Them)

Retrieval Misses

The document exists in your corpus but wasn't retrieved. Debug by running the query embedding against the target document's embedding directly — if the similarity is low, the problem is in your chunking or embedding model. If the similarity is high but the document wasn't in the top-K, your index may have quantization issues or you're not retrieving enough candidates before re-ranking.

Context Poisoning

You retrieved 10 chunks, but 7 of them are irrelevant. The LLM now has to distinguish signal from noise, and it doesn't always succeed. The fix is upstream: better chunking, better ranking, and aggressive relevance thresholds. Drop any chunk below a minimum similarity score rather than always returning a fixed K.

Hallucination Despite Correct Context

The right chunk was retrieved and included in the prompt, but the LLM still hallucinated. This is often a prompt engineering problem. Explicit instructions like "Answer based ONLY on the provided context. If the context doesn't contain the answer, say so" are essential, not optional. Also consider: is the relevant information buried in a long passage? The "lost in the middle" effect applies within individual chunks too.

Stale Embeddings

Your documents were updated but the embeddings weren't re-computed. This is the RAG equivalent of a cache invalidation bug. Build your indexing pipeline with incremental updates from day one. Track document hashes and re-embed only what changed. At scale, a full re-index is a multi-hour, multi-GPU operation — you don't want to do it unnecessarily.

Lessons from Scale

What changes when you go from a demo to a production system serving millions of users?

Index management becomes a first-class concern. You need index versioning, blue-green deployments for index updates, and the ability to roll back a bad index without downtime. Your index is as critical as your database — treat it that way.

Latency budgets force hard tradeoffs. At Microsoft-scale, every millisecond matters. You might skip re-ranking on low-importance queries. You might use a smaller embedding model for initial retrieval and reserve the expensive cross-encoder for the final top-10. Tiered retrieval architectures are common in production.

Monitoring is non-negotiable. Track retrieval precision and recall against labeled query sets. Monitor embedding drift over time. Alert on sudden drops in answer quality. Log the full pipeline: query → retrieved chunks → generated answer, so you can debug failures post-hoc. The RAG pipeline that you can't observe is the RAG pipeline that silently degrades.

Evaluation is continuous. Build evaluation sets that reflect your actual query distribution. Automated metrics (faithfulness, relevance, answer correctness) run on every pipeline change. Human evaluation catches what automated metrics miss. This isn't optional — it's how you maintain quality over time.

Conclusion

The RAG pipeline is deceptively simple to prototype and genuinely hard to operate at scale. The architecture diagram fits on a napkin: embed, retrieve, generate. But the difference between a demo and a production system lives in the details — how you chunk documents, which embedding model you choose, how you rank and filter results, how you manage the context window, and how you monitor the whole thing.

My advice: build incrementally. Start with the simplest version that works, instrument everything, and let your evaluation data tell you where to invest next. Don't over-engineer the retrieval before you've verified your chunking is sound. Don't add re-ranking before you've confirmed your base retrieval is reasonable.

And don't skip the boring parts. Chunking and ranking aren't glamorous, but they're where production RAG systems are won or lost. The LLM is the easy part.

Nrk Raju Guthikonda is a Senior Software Engineer at Microsoft on the Copilot Search Infrastructure team, where he works on Semantic Indexing and Retrieval-Augmented Generation. He has built 116+ open-source repositories spanning AI/ML, healthcare, developer tools, and creative AI. Find his work on GitHub at github.com/kennedyraju55.