Forem: Ganesh Navale

How to Stream AI Responses from Rails API to React (SSE Magic ✨)

Ganesh Navale — Sat, 31 Jan 2026 09:40:01 +0000

🤔 Ever wonder how to stream AI responses to React from Rails?

You know that feeling when your users click "Improve with AI" and stare at a loading spinner for 30 seconds? Yeah, not great UX.

Let me show you how to make AI responses appear word-by-word like ChatGPT, without the complexity of WebSockets.

👉 The Problem

Traditional approach:

React app → Rails API → OpenAI → Wait 30 seconds → Full response

What user sees: Loading spinner forever 😴

The user experience:

Clicks button
Waits... and waits...
Wonders if it's broken
Considers closing the tab

✨ The Solution: Server-Sent Events (SSE)

Modern approach:

React app → Rails SSE endpoint → Streams tokens → Real-time updates

What user sees: AI typing word-by-word, just like ChatGPT ✨

Why SSE?

✅ Simpler than WebSockets
✅ Works over standard HTTP
✅ Native browser streaming support (ReadableStream)
✅ Perfect for one-way streaming (server → client)

🛠️ Let's Build It

Step 1: Rails API Backend

First, create the SSE helper:

# app/services/sse.rb
class SSE
  def initialize(stream)
    @stream = stream
  end

  def write(data, event: nil)
    @stream.write("event: #{event}\n") if event
    @stream.write("data: #{data.to_json}\n\n")
  end

  def close
    @stream.close
  end
end

Step 2: Create the Controller

# app/controllers/api/v1/ai_improvements_controller.rb
class Api::V1::AiImprovementsController < ApplicationController
  include ActionController::Live

  def create
    response.headers['Content-Type'] = 'text/event-stream'
    response.headers['X-Accel-Buffering'] = 'no' # Disable nginx buffering
    response.headers['Access-Control-Allow-Origin'] = ENV['FRONTEND_URL']

    sse = SSE.new(response.stream)
    content = params[:content]

    # Your AI service that streams tokens
    AiContentImprover.new(content).improve_streaming do |token|
      sse.write({ token: token }, event: 'token')
    end

    sse.write({ done: true }, event: 'complete')

  rescue IOError
    # Client disconnected - this is normal
  ensure
    sse.close
  end
end

Step 3: AI Service with Streaming

# app/services/ai_content_improver.rb
class AiContentImprover
  def initialize(content)
    @content = content
    @client = OpenAI::Client.new(access_token: ENV['OPENAI_API_KEY'])
  end

  def improve_streaming(&block)
    @client.chat(
      parameters: {
        model: "gpt-4-turbo-preview",
        messages: [
          { role: "system", content: "Improve this content. Make it clearer and more engaging." },
          { role: "user", content: @content }
        ],
        stream: proc do |chunk, _bytesize|
          text = chunk.dig("choices", 0, "delta", "content")
          block.call(text) if text
        end
      }
    )
  end
end

Step 4: Routes

# config/routes.rb
Rails.application.routes.draw do
  namespace :api do
    namespace :v1 do
      post 'ai_improvements', to: 'ai_improvements#create'
    end
  end
end

⚛️ React Frontend

Now the fun part - consuming the SSE stream in React:

import { useState, useRef, useEffect } from 'react'

function AiAssistant({ content }) {
  const [suggestion, setSuggestion] = useState('')
  const [loading, setLoading] = useState(false)
  const abortControllerRef = useRef(null)

  // Cleanup on unmount
  useEffect(() => {
    return () => {
      if (abortControllerRef.current) {
        abortControllerRef.current.abort()
      }
    }
  }, [])

  const improveContent = async () => {
    // Cancel any existing request
    if (abortControllerRef.current) {
      abortControllerRef.current.abort()
    }

    setLoading(true)
    setSuggestion('')
    abortControllerRef.current = new AbortController()

    try {
      const API_URL = 'http://localhost:3000'
      const response = await fetch(`${API_URL}/api/v1/ai_improvements`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Accept': 'text/event-stream',
        },
        body: JSON.stringify({ content }),
        signal: abortControllerRef.current.signal,
      })

      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`)
      }

      const reader = response.body.getReader()
      const decoder = new TextDecoder()
      let buffer = ''

      while (true) {
        const { value, done } = await reader.read()
        if (done) break

        buffer += decoder.decode(value, { stream: true })
        const lines = buffer.split('\n')
        buffer = lines.pop() || ''

        for (const line of lines) {
          if (!line.startsWith('data: ')) continue

          const data = JSON.parse(line.slice(6))
          if (data.token) {
            setSuggestion(prev => prev + data.token)
          }
        }
      }
    } catch (error) {
      if (error.name !== 'AbortError') {
        console.error('Stream error:', error)
      }
    } finally {
      setLoading(false)
      abortControllerRef.current = null
    }
  }

  return (
    <div className="ai-assistant">
      <button 
        onClick={improveContent} 
        disabled={loading}
        className="improve-btn"
      >
        {loading ? '✨ AI thinking...' : '✨ Improve with AI'}
      </button>

      {suggestion && (
        <div className="suggestion-box">
          <p>{suggestion}</p>
        </div>
      )}
    </div>
  )
}

export default AiAssistant

🎨 Add Some Style

.ai-assistant {
  margin: 20px 0;
}

.improve-btn {
  background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
  color: white;
  border: none;
  padding: 12px 24px;
  border-radius: 8px;
  cursor: pointer;
  font-weight: 600;
  transition: transform 0.2s;
}

.improve-btn:hover {
  transform: translateY(-2px);
}

.improve-btn:disabled {
  opacity: 0.6;
  cursor: not-allowed;
}

.suggestion-box {
  margin-top: 16px;
  padding: 16px;
  background: #f7fafc;
  border-left: 4px solid #667eea;
  border-radius: 4px;
  line-height: 1.6;
  white-space: pre-wrap;
}

🎉 The Magic Happens

When the user clicks "✨ Improve with AI":

Browser creates connection using fetch with ReadableStream
Rails streams AI tokens one-by-one
JavaScript appends each token to the UI
User sees AI "typing" in real-time
Connection closes automatically when done

Result:

✨ Token-by-token streaming
📱 Native browser API (no library needed)
🔄 Easy to add retry logic if needed
🚀 Much cleaner than WebSockets for this use case

🔥 Advanced: Add Rate Limiting

Prevent users from spamming your expensive AI endpoint:

# app/controllers/api/v1/ai_improvements_controller.rb
before_action :check_rate_limit

def check_rate_limit
  key = "ai_improve:#{current_user.id}"
  count = Rails.cache.increment(key, 1, expires_in: 1.minute)

  return unless count > 5

  sse = SSE.new(response.stream)
  sse.write({ error: "⏸️ Slow down! Try again in a minute." }, event: 'error')
  sse.close
end

Update React to handle errors:

// In the stream parsing loop
if (data.error) {
  alert(data.error)
  return
}

💰 Bonus: Cost Tracking

Track your AI usage to avoid surprise bills:

# app/models/ai_usage.rb
class AiUsage < ApplicationRecord
  belongs_to :user

  def self.total_cost_today
    where('created_at >= ?', Time.zone.today).sum(:estimated_cost)
  end
end

# In your controller:
usage = AiUsage.create!(
  user: current_user,
  action: 'content_improvement',
  input_tokens: content.length / 4, # rough estimate
)

# After streaming completes:
usage.update!(output_tokens: token_count, estimated_cost: calculate_cost)

💡 Tip: For production, use the rack-cors gem instead of inline headers to properly handle preflight requests.

📊 SSE vs WebSocket vs Polling

Feature	SSE (fetch)	WebSocket	Long Polling
Setup Complexity	⭐ Simple	⭐⭐⭐ Complex	⭐⭐ Moderate
HTTP Methods	✅ All (GET, POST)	N/A	✅ All
Custom Headers	✅ Full control	⚠️ Limited	✅ Full control
Bidirectional	❌ Server → Client	✅ Both ways	❌ Client → Server
Auto Reconnect	❌ Manual	❌ Manual	N/A
Connection Overhead	Low (single HTTP)	Low (persistent)	High (repeated requests)
Scalability	⭐⭐ Good	⭐⭐⭐ Excellent	⭐ Poor
Proxy/Firewall Friendly	✅ Yes (HTTP)	⚠️ Sometimes blocked	✅ Yes
Browser Support	✅ All modern	✅ All modern	✅ All
Best For	AI streaming, notifications	Real-time chat, gaming	Legacy systems

For AI streaming: SSE wins! ✨

Why? AI responses are inherently one-directional (server → client), making WebSocket's bidirectional capability unnecessary overhead. SSE with fetch gives you POST support for sending prompts while streaming responses back.

🚀 What's Next?

Want to level up? Try:

Add Anthropic Claude streaming (different API, same concept)
Implement retry logic for failed streams
Add progress indicators showing % complete
Cache similar requests to save costs
Support multiple AI models (let users choose)

💬 Over to You

Have you used SSE before? How was your experience?
Would you choose SSE or WebSockets for AI streaming?
What other AI features are you building in your Rails + React apps?

Drop a comment below! 👇

🔗 Useful Resources

Found this helpful? Give it a ❤️ and follow me for more Rails + React + AI tips!

Happy coding! 🚀

rails #react #ai #sse #openai #streaming #webdev

How to Stream AI Responses from Rails API to React (SSE Magic ✨)

Ganesh Navale — Sat, 31 Jan 2026 09:20:38 +0000

🤔 Ever wonder how to stream AI responses to React from Rails?

You know that feeling when your users click "Improve with AI" and stare at a loading spinner for 30 seconds? Yeah, not great UX.

Let me show you how to make AI responses appear word-by-word like ChatGPT, without the complexity of WebSockets.

👉 The Problem

Traditional approach:

React app → Rails API → OpenAI → Wait 30 seconds → Full response

What user sees: Loading spinner forever 😴

The user experience:

Clicks button
Waits... and waits...
Wonders if it's broken
Considers closing the tab

✨ The Solution: Server-Sent Events (SSE)

Modern approach:

React app → Rails SSE endpoint → Streams tokens → Real-time updates

What user sees: AI typing word-by-word, just like ChatGPT ✨

Why SSE?

✅ Simpler than WebSockets
✅ Works over standard HTTP
✅ Native browser streaming support (ReadableStream)
✅ Perfect for one-way streaming (server → client)

🛠️ Let's Build It

Step 1: Rails API Backend

First, create the SSE helper:

# app/services/sse.rb
class SSE
  def initialize(stream)
    @stream = stream
  end

  def write(data, event: nil)
    @stream.write("event: #{event}\n") if event
    @stream.write("data: #{data.to_json}\n\n")
  end

  def close
    @stream.close
  end
end

Step 2: Create the Controller

# app/controllers/api/v1/ai_improvements_controller.rb
class Api::V1::AiImprovementsController < ApplicationController
  include ActionController::Live

  def create
    response.headers['Content-Type'] = 'text/event-stream'
    response.headers['X-Accel-Buffering'] = 'no' # Disable nginx buffering
    response.headers['Access-Control-Allow-Origin'] = ENV['FRONTEND_URL']

    sse = SSE.new(response.stream)
    content = params[:content]

    # Your AI service that streams tokens
    AiContentImprover.new(content).improve_streaming do |token|
      sse.write({ token: token }, event: 'token')
    end

    sse.write({ done: true }, event: 'complete')

  rescue IOError
    # Client disconnected - this is normal
  ensure
    sse.close
  end
end

Step 3: AI Service with Streaming

# app/services/ai_content_improver.rb
class AiContentImprover
  def initialize(content)
    @content = content
    @client = OpenAI::Client.new(access_token: ENV['OPENAI_API_KEY'])
  end

  def improve_streaming(&block)
    @client.chat(
      parameters: {
        model: "gpt-4-turbo-preview",
        messages: [
          { role: "system", content: "Improve this content. Make it clearer and more engaging." },
          { role: "user", content: @content }
        ],
        stream: proc do |chunk, _bytesize|
          text = chunk.dig("choices", 0, "delta", "content")
          block.call(text) if text
        end
      }
    )
  end
end

Step 4: Routes

# config/routes.rb
Rails.application.routes.draw do
  namespace :api do
    namespace :v1 do
      post 'ai_improvements', to: 'ai_improvements#create'
    end
  end
end

⚛️ React Frontend

Now the fun part - consuming the SSE stream in React:

import { useState, useRef, useEffect } from 'react'

function AiAssistant({ content }) {
  const [suggestion, setSuggestion] = useState('')
  const [loading, setLoading] = useState(false)
  const abortControllerRef = useRef(null)

  // Cleanup on unmount
  useEffect(() => {
    return () => {
      if (abortControllerRef.current) {
        abortControllerRef.current.abort()
      }
    }
  }, [])

  const improveContent = async () => {
    // Cancel any existing request
    if (abortControllerRef.current) {
      abortControllerRef.current.abort()
    }

    setLoading(true)
    setSuggestion('')
    abortControllerRef.current = new AbortController()

    try {
      const API_URL = 'http://localhost:3000'
      const response = await fetch(`${API_URL}/api/v1/ai_improvements`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Accept': 'text/event-stream',
        },
        body: JSON.stringify({ content }),
        signal: abortControllerRef.current.signal,
      })

      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`)
      }

      const reader = response.body.getReader()
      const decoder = new TextDecoder()
      let buffer = ''

      while (true) {
        const { value, done } = await reader.read()
        if (done) break

        buffer += decoder.decode(value, { stream: true })
        const lines = buffer.split('\n')
        buffer = lines.pop() || ''

        for (const line of lines) {
          if (!line.startsWith('data: ')) continue

          const data = JSON.parse(line.slice(6))
          if (data.token) {
            setSuggestion(prev => prev + data.token)
          }
        }
      }
    } catch (error) {
      if (error.name !== 'AbortError') {
        console.error('Stream error:', error)
      }
    } finally {
      setLoading(false)
      abortControllerRef.current = null
    }
  }

  return (
    <div className="ai-assistant">
      <button 
        onClick={improveContent} 
        disabled={loading}
        className="improve-btn"
      >
        {loading ? '✨ AI thinking...' : '✨ Improve with AI'}
      </button>

      {suggestion && (
        <div className="suggestion-box">
          <p>{suggestion}</p>
        </div>
      )}
    </div>
  )
}

export default AiAssistant

🎨 Add Some Style

.ai-assistant {
  margin: 20px 0;
}

.improve-btn {
  background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
  color: white;
  border: none;
  padding: 12px 24px;
  border-radius: 8px;
  cursor: pointer;
  font-weight: 600;
  transition: transform 0.2s;
}

.improve-btn:hover {
  transform: translateY(-2px);
}

.improve-btn:disabled {
  opacity: 0.6;
  cursor: not-allowed;
}

.suggestion-box {
  margin-top: 16px;
  padding: 16px;
  background: #f7fafc;
  border-left: 4px solid #667eea;
  border-radius: 4px;
  line-height: 1.6;
  white-space: pre-wrap;
}

🎉 The Magic Happens

When the user clicks "✨ Improve with AI":

Browser creates connection using fetch with ReadableStream
Rails streams AI tokens one-by-one
JavaScript appends each token to the UI
User sees AI "typing" in real-time
Connection closes automatically when done

Result:

✨ Token-by-token streaming
📱 Native browser API (no library needed)
🔄 Easy to add retry logic if needed
🚀 Much cleaner than WebSockets for this use case

🔥 Advanced: Add Rate Limiting

Prevent users from spamming your expensive AI endpoint:

# app/controllers/api/v1/ai_improvements_controller.rb
before_action :check_rate_limit

def check_rate_limit
  key = "ai_improve:#{current_user.id}"
  count = Rails.cache.increment(key, 1, expires_in: 1.minute)

  return unless count > 5

  sse = SSE.new(response.stream)
  sse.write({ error: "⏸️ Slow down! Try again in a minute." }, event: 'error')
  sse.close
end

Update React to handle errors:

// In the stream parsing loop
if (data.error) {
  alert(data.error)
  return
}

💰 Bonus: Cost Tracking

Track your AI usage to avoid surprise bills:

# app/models/ai_usage.rb
class AiUsage < ApplicationRecord
  belongs_to :user

  def self.total_cost_today
    where('created_at >= ?', Time.zone.today).sum(:estimated_cost)
  end
end

# In your controller:
usage = AiUsage.create!(
  user: current_user,
  action: 'content_improvement',
  input_tokens: content.length / 4, # rough estimate
)

# After streaming completes:
usage.update!(output_tokens: token_count, estimated_cost: calculate_cost)

💡 Tip: For production, use the rack-cors gem instead of inline headers to properly handle preflight requests.

📊 SSE vs WebSocket vs Polling

Feature	SSE (fetch)	WebSocket	Long Polling
Setup Complexity	⭐ Simple	⭐⭐⭐ Complex	⭐⭐ Moderate
HTTP Methods	✅ All (GET, POST)	N/A	✅ All
Custom Headers	✅ Full control	⚠️ Limited	✅ Full control
Bidirectional	❌ Server → Client	✅ Both ways	❌ Client → Server
Auto Reconnect	❌ Manual	❌ Manual	N/A
Connection Overhead	Low (single HTTP)	Low (persistent)	High (repeated requests)
Scalability	⭐⭐ Good	⭐⭐⭐ Excellent	⭐ Poor
Proxy/Firewall Friendly	✅ Yes (HTTP)	⚠️ Sometimes blocked	✅ Yes
Browser Support	✅ All modern	✅ All modern	✅ All
Best For	AI streaming, notifications	Real-time chat, gaming	Legacy systems

For AI streaming: SSE wins! ✨

🚀 What's Next?

Want to level up? Try:

Add Anthropic Claude streaming (different API, same concept)
Implement retry logic for failed streams
Add progress indicators showing % complete
Cache similar requests to save costs
Support multiple AI models (let users choose)

💬 Over to You

Have you used SSE before? How was your experience?
Would you choose SSE or WebSockets for AI streaming?
What other AI features are you building in your Rails + React apps?

Drop a comment below! 👇

🔗 Useful Resources

Found this helpful? Give it a ❤️ and follow me for more Rails + React + AI tips!

Happy coding! 🚀

rails #react #ai #sse #openai #streaming #webdev

RAG vs Fine-Tuning: Choosing the Right Approach for Your LLM Application

Ganesh Navale — Sat, 24 Jan 2026 08:28:57 +0000

When building applications with Large Language Models (LLMs), you'll often face a critical decision: should you use Retrieval-Augmented Generation (RAG) or fine-tune your model? Both approaches can enhance your LLM's performance, but they solve different problems and come with distinct trade-offs.

What is RAG?

Retrieval-Augmented Generation is a technique that combines an LLM with an external knowledge base. When a user asks a question, the system first retrieves relevant information from a database or document collection, then feeds that context to the LLM alongside the query.

How RAG Works:

User submits a query
System converts query into embeddings
Relevant documents are retrieved from a vector database
Retrieved context is added to the prompt
LLM generates a response using both the query and retrieved context

Example Use Case:
A customer support chatbot that needs to answer questions based on your company's latest documentation, product updates, and knowledge base articles.

What is Fine-Tuning?

Fine-tuning involves training an existing LLM on a specialized dataset to adapt its behavior, writing style, or domain knowledge. You're essentially teaching the model new patterns by updating its weights through additional training.

How Fine-Tuning Works:

Prepare a dataset of input-output pairs
Initialize from a pre-trained model
Train the model on your custom dataset
Adjust hyperparameters and validate performance
Deploy the customized model

Example Use Case:
A medical diagnosis assistant that needs to understand specialized medical terminology and respond with the appropriate clinical tone and precision.

Key Differences at a Glance

Aspect	RAG	Fine-Tuning
Knowledge Updates	Easy - just update the database	Requires retraining the model
Cost	Lower - no model training needed	Higher - GPU costs for training
Setup Time	Fast - days to implement	Slower - weeks of preparation and training
Accuracy	Source-attributable and verifiable	Absorbed into model weights
Data Requirements	Can work with small datasets	Needs substantial training data
Maintenance	Ongoing content management	Periodic retraining cycles

When to Use RAG

RAG is your best choice when:

Dynamic Knowledge Requirements: Your information changes frequently. If you're working with news, documentation, or any rapidly updating content, RAG allows you to update your knowledge base without retraining.

Source Attribution Matters: You need to cite sources or show users where information came from. RAG naturally provides this since it retrieves specific documents.

Limited Budget: You want to avoid the computational costs of training. RAG works with off-the-shelf models and standard infrastructure.

Quick Iteration: You need to get something working fast and iterate based on user feedback. RAG systems can be set up and modified much more quickly.

Large, Diverse Knowledge Bases: You're working with extensive documentation where the model would struggle to memorize everything through fine-tuning.

When to Use Fine-Tuning

Fine-tuning makes sense when:

Behavioral Consistency: You need the model to consistently follow specific formats, tones, or styles. Fine-tuning can make these behaviors more reliable than prompting alone.

Domain-Specific Language: Your field uses specialized terminology, syntax, or reasoning patterns that general models don't handle well. Medical, legal, or technical domains often benefit from fine-tuning.

Reduced Latency: You want faster responses without the overhead of retrieval. Fine-tuned knowledge is baked into the model.

Proprietary Knowledge: Your training data contains sensitive information that shouldn't be stored in a retrievable database.

Task-Specific Performance: You're optimizing for a narrow, well-defined task where fine-tuning can significantly boost accuracy.

Can You Use Both?

Absolutely! Many production systems combine RAG and fine-tuning for optimal results:

Fine-tune for style, tone, and domain language
Use RAG for factual, up-to-date information

This hybrid approach gives you the best of both worlds - a model that speaks your domain's language while staying current with the latest information.

Example Architecture:

User Query → Fine-tuned LLM (understands domain) 
          → RAG System (retrieves latest facts)
          → Combined Response

Practical Decision Framework

Ask yourself these questions:

How often does your knowledge change?
- Daily/Weekly → RAG
- Rarely → Fine-tuning
What's your primary goal?
- Access to information → RAG
- Behavioral modification → Fine-tuning
What's your budget?
- Limited → RAG
- Substantial → Consider fine-tuning
Do you need source citations?
- Yes → RAG
- No → Either works
How much training data do you have?
- Limited (<1000 examples) → RAG
- Substantial (>10,000 examples) → Fine-tuning possible

Real-World Examples

RAG Success Story: A legal tech company built a contract analysis tool using RAG. They maintain a database of legal precedents and regulations that updates weekly. RAG allows them to incorporate new rulings immediately without retraining, and they can show lawyers exactly which precedents informed each analysis.

Fine-Tuning Success Story: A healthcare startup fine-tuned a model on thousands of medical conversations to handle patient triage. The model learned to ask the right follow-up questions and use appropriate medical terminology, providing a consistent experience that RAG alone couldn't achieve.

Getting Started

Starting with RAG:

Choose a vector database (Pinecone, Weaviate, Chroma)
Prepare your documents and create embeddings
Implement retrieval logic
Integrate with your LLM API
Test and refine your retrieval strategy

Starting with Fine-Tuning:

Collect and clean your training data
Format as input-output pairs
Choose a base model and platform (OpenAI, Hugging Face)
Train with proper validation
Evaluate on held-out test data
Deploy and monitor

Conclusion

Neither RAG nor fine-tuning is universally better - they excel in different scenarios. RAG shines when you need flexibility, quick updates, and source attribution. Fine-tuning wins when you need behavioral consistency, domain expertise, and reduced latency.

For many production applications, a hybrid approach leveraging both techniques provides the most robust solution. Start with RAG for its speed and flexibility, then consider fine-tuning if you identify specific behavioral improvements that prompting can't solve.

The key is understanding your specific requirements, constraints, and goals. With this framework, you can make an informed decision that sets your LLM application up for success.

What's your experience with RAG and fine-tuning? Have you found one more effective than the other for your use cases? Let me know in the comments!

Why My Second RAG System Was Built in Rails, Not Python’s FastAPI

Ganesh Navale — Tue, 20 Jan 2026 15:43:30 +0000

Built the same RAG system in FastAPI and Ruby on Rails. FastAPI took weeks, Rails took 24 hours. Here's what that taught me about choosing frameworks for AI products.

A hands-on comparison from a Rails developer's first AI project

Picture this: You’re starting a new RAG project. You open your laptop and wonder: “Do I really need Python for this?”

Every tutorial assumes Python. Almost every example uses FastAPI or Flask. And there I was thinking, “But… I already know Rails.”

I’m a Ruby on Rails developer, and I love Rails. I decided to dive in anyway and build a RAG system with FastAPI, learning embeddings, vector databases, and how to make an LLM actually answer questions from documents.

Then our company announced an AI hackathon. I had 48 hours to build another RAG system. This time, I built it in Rails. It wasn’t about rewriting; it was a practical choice under tight deadlines.

Same features. Same vector database. Same LLM. Same project, different framework.

What the RAG system actually does

This RAG system is essentially a question-answering engine over your own documents. A user uploads PDFs or text, the system embeds and stores them, and when a query is submitted it finds the most relevant content and uses an LLM to answer based on that retrieved context.
For example, instead of asking a model “What is the capital of France?” — which it might already know — the system answers specific questions about a custom set of documents that the user has uploaded

This article shares my experience building the same RAG system twice, what changed, and what I learned about Rails, FastAPI, and building AI-powered features in real projects.

TL;DR

Built the same RAG system in FastAPI (took weeks) and Rails (took 24 hours)
The actual AI logic was identical—the difference was infrastructure
Rails' mature tooling (Sidekiq, ActiveRecord, console) made development faster
Python still wins for ML-heavy experimentation
My recommendation: Rails for the app, Python microservices only when needed
You don't need to rewrite your Rails app in Python to add AI features

What I built (in both versions)

Both RAG systems had the same core functionality:

The tech stack:

FastAPI version:

FastAPI for the API layer
Celery for background jobs
SQLAlchemy for database access
OpenAI API for embeddings and completions
pgvector for similarity search

Rails version:

Rails API backend
React frontend
Sidekiq for background jobs
ActiveRecord for database access
Same OpenAI API and pgvector setup

The AI logic was identical. The framework wrapping it was different.

The Architecture (Same for Both)

Both implementations follow this exact flow:

Document Upload → User uploads PDF via web interface
Text Extraction → Extract and clean text from PDF
Chunking → Split document into ~500 token chunks with overlap
Embedding Generation → Send chunks to OpenAI's embedding API
Vector Storage → Store embeddings in Pinecone with metadata
Question Processing → When user asks a question:
- Generate embedding for the question(text-embedding-3-small model)
- Query Pinecone for top 5 similar chunks
- Send question + context to GPT-4
- Stream response back to user

Next Up: Where the Differences Actually Showed Up

The FastAPI Version: Where I Spent My Time

Building the FastAPI version worked. But I spent more time on infrastructure than on the actual AI features.

In theory, Celery handles async tasks. In practice, I became the one handling Celery.

When an embedding job failed (and they did, API timeouts, rate limits, malformed PDFs), here's what debugging looked like:

Background jobs became a maintenance burden:

# FastAPI/Celery - Replay a failed embedding job
# 1. Find the task ID in logs
# 2. Check Celery flower or redis
# 3. Manually construct retry logic
# 4. Hope the async session doesn't break again

@celery_app.task(bind=True, max_retries=3)
async def embed_document(self, doc_id):
    try:
        async with get_db_session() as session:
            # embedding logic
            pass
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

Compare this to Rails:

# Rails - Replay failed job from console or UI
DocumentEmbeddingJob.perform_later(document.id)

# Built-in retry with exponential backoff
class DocumentEmbeddingJob < ApplicationJob
  retry_on OpenAI::Error, wait: :exponentially_longer, attempts: 5

  def perform(document_id)
    document = Document.find(document_id)
    embedding = OpenAI.embed(document.content)
    document.update!(embedding: embedding)
  end
end

In Rails, I can see failed jobs in Sidekiq's web UI, click "Retry," and watch it work. In FastAPI, I was writing custom monitoring and retry logic.

Database session management became a daily puzzle:

Every async endpoint needed careful session handling. I'd write a feature, run tests, and watch them randomly fail because some session somewhere wasn't properly closed. I spent more time reading asyncio documentation than building features.

# FastAPI - Manual session lifecycle everywhere
@app.post("/documents")
async def create_document(doc: DocumentCreate):
    async with get_db_session() as session:
        async with session.begin():
            # Don't forget to close this!
            # Or rollback on error!
            # Or handle connection pool limits!
            pass

Meanwhile in Rails? ActiveRecord just handles it. I never thought about sessions once during the hackathon.

Deployment felt fragile:

I had to manually configure:

Async workers
Job queues
Retry policies
Monitoring dashboards
Database connection pools for async

Rails gives me all of this out of the box.

The Rails version: exactly what I needed

During the hackathon, I had 48 hours to ship a working demo. Not a prototype. Not a proof-of-concept. A working system that non-technical people could use.

I chose Rails not because it's "better for AI" (it's probably not), but because I knew exactly where my time would go: building features, not configuring infrastructure.

Here's what the system did:

Ingested documentation from multiple sources
Split content into chunks
Generated embeddings via OpenAI
Stored vectors in Postgres with pgvector
Answered questions using retrieved context
Actually ship it before the demo

The entire backend:

# Model
class Document < ApplicationRecord
  has_neighbors :embedding

  after_create_commit :enqueue_embedding_job

  def enqueue_embedding_job
    DocumentEmbeddingJob.perform_later(id)
  end
end

# Background job
class DocumentEmbeddingJob < ApplicationJob
  retry_on OpenAI::Error, wait: :exponentially_longer

  def perform(document_id)
    document = Document.find(document_id)

    chunks = split_into_chunks(document.content)
    chunks.each do |chunk|
      embedding = generate_embedding(chunk)
      DocumentChunk.create!(
        document: document,
        content: chunk,
        embedding: embedding
      )
    end
  end

  private

  def generate_embedding(text)
    client = OpenAI::Client.new
    response = client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )
    response.dig("data", 0, "embedding")
  end
end

# Query service
class RagQueryService
  def initialize(query)
    @query = query
    @embedding = generate_embedding(query) # Convert question to vector
  end

  def answer
    # Find the 5 most similar document chunks
    relevant_chunks = DocumentChunk
      .nearest_neighbors(:embedding, @embedding, distance: "cosine")
      .limit(5)

    # Combine them into context
    context = relevant_chunks.map(&:content).join("\n\n")

    # Ask GPT-4 with the context
    client = OpenAI::Client.new
    response = client.chat(
      parameters: {
        model: "gpt-4",
        messages: [
          { role: "system", content: "Answer based on this context: #{context}" },
          { role: "user", content: @query }
        ]
      }
    )

    response.dig("choices", 0, "message", "content")
  end
end

Reality check: This code is simplified for the article. The real version had error handling, logging, and rate limiting. But the core logic? Pretty much this.

That's it. The entire RAG pipeline in ~60 lines of readable Ruby.
No async session management. No custom retry logic. No Celery flower dashboard. Just Rails doing what Rails does best: letting you build features instead of infrastructure.

Could I have made the FastAPI version this clean? Maybe. But I didn't have time to figure it out. And that's the point.

What made Rails faster for me

I shipped the Rails version in 24 hours. The FastAPI version took weeks to get stable.

Here's why:

1. Background jobs are a solved problem

Sidekiq gives me:

Web UI to monitor jobs
Automatic retries with backoff
Dead job queues
Performance metrics
One-click job replay

I didn't write any of this. It was already there.

2. Database access is predictable

No async session managers. No connection pool tuning. No event loop surprises. It just works.

3. Debugging is straightforward

When an embedding job failed:

Open Sidekiq UI
See the error and full backtrace
Click "Retry"
Check logs if needed

In FastAPI, I was tailing Celery logs and rebuilding context manually.

4. The ecosystem has what I needed

ruby-openai gem for API calls
neighbor gem for vector similarity
pgvector extension for Postgres
Standard Rails patterns for everything else

No async complications. No compatibility issues.

5. The developer workflow is integrated, not assembled

This is easy to underestimate until you feel it.

Rails gives you a tight feedback loop by default:

A powerful interactive Rails console for debugging live data and jobs
Database migrations that are simple, versioned, and reversible
A test setup that’s predictable and deeply integrated with the framework

When I wanted to:

Inspect a document’s embeddings
Replay a failed job
Tweak a schema and re-run ingestion
Write or debug a failing test

When I needed to inspect embeddings, replay a failed job, or tweak the schema, I did it from the console in seconds.

In the FastAPI setup, these same tasks required more manual work:

Managing Alembic migrations explicitly
Configuring async test fixtures
Debugging through logs instead of an interactive console

None of this is impossible in Python — but it is more fragmented.

Rails optimizes for flow.
FastAPI optimizes for flexibility.

When you're iterating on AI features under a deadline, that difference compounds daily.

The key insight: AI primitives are framework-agnostic

After building both versions, here's what became clear:

The actual AI logic was identical.

Both systems used the same process:

Split documents into chunks
Generate embeddings via OpenAI API
Store vectors in Postgres
Retrieve similar chunks
Pass context to LLM

The intelligence came from:

Quality of input data
Chunking strategy
Prompt engineering
Retrieval precision

Not from the framework.

Rails didn’t make the model smarter. It made the system easier to reason about, operate, and change.

And for a product engineer shipping features, that matters more than access to the latest ML libraries.

Where Python still clearly wins

Let me be clear: there are cases where Python is the right choice.

Use Python when you need:

Advanced document loaders (LangChain, LlamaIndex)
Custom re-ranking models
Sophisticated evaluation frameworks
Rapid ML experimentation
Fine-tuning workflows

For research and ML-heavy work, Python is unmatched.

Examples where I'd choose Python:

You're experimenting with multiple embedding models weekly
You need custom re-ranking with a BERT model
You're running A/B tests on different chunking strategies
Your team is already Python-first

But for building a production RAG feature in an existing Rails app? You probably don't need to rewrite everything in Python.

My Approach Now: Hybrid Architecture

After building both versions, I use this mental model:

Rails handles the application:

API endpoints
Background jobs
Database models
User authentication
Business logic

Python microservices for ML-specific work:

Custom re-ranking models
Advanced document parsing
Specialized ML pipelines
Evaluation frameworks

Why this works:

Product code stays stable and maintainable
AI experiments stay isolated
Infrastructure stays simple
Team velocity stays high

Instead of migrating my entire Rails app to FastAPI, I integrate Python only where it adds specific value.

The real takeaway

Strip away the AI layer, and a RAG system is still just a distributed application:

Background jobs
Database transactions
Retries and failures
User-facing latency

The framework you choose determines how painful these problems are to live with.

That's why this comparison isn't really about Rails vs FastAPI.

It's about choosing tools that let you focus on product behavior instead of infrastructure glue.

Final thoughts

Building the same RAG system in two different frameworks taught me something simple but important:

The hard parts of production software aren't the AI API calls. They're:

Reliable background processing
Debugging production failures
Managing deployments
Maintaining clean architecture
Shipping quickly

If you're already working in Rails (or Django, or any mature web framework), you already have solutions for these problems. Adding AI features doesn't change that.

Python has incredible AI tooling. Rails has incredible application tooling.

You don't need to choose one or the other. You can use both strategically.

If you're a Rails developer wondering whether you need to learn FastAPI to build AI features: you probably don't. Start with Rails. Add Python services only when you hit a real limitation.

Building AI features in non-Python frameworks? I'd love to hear about your experience. Drop a comment or connect with me on LinkedIn.