Forem: Abhi

Real-Time Data Streaming Made Simple: Server-Sent Events in Ballerina

Abhi — Thu, 30 Oct 2025 22:20:45 +0000

Introduction

In today's world of real-time applications—from live sports scores to stock tickers, chat notifications to IoT dashboards—the ability to push data from server to client is no longer a luxury, it's a necessity. While WebSockets often steal the spotlight, there's an elegant, simpler alternative that's perfect for many use cases: Server-Sent Events (SSE).

In this post, we'll explore how Ballerina, with its cloud-native design and powerful HTTP module, makes implementing SSE incredibly straightforward. We'll build real working examples, understand the patterns, and see why SSE might be exactly what your next project needs.

What Are Server-Sent Events?

Server-Sent Events (SSE) is a web technology that enables servers to push data to web clients over a single, long-lived HTTP connection. Think of it as a one-way communication channel from your server to the browser—perfect for scenarios where the server needs to continuously update the client.

The SSE Advantage

Why choose SSE over WebSockets?

✅ Simpler to implement - Built on standard HTTP, no special protocols

✅ Automatic reconnection - Browsers handle reconnection automatically

✅ Event IDs & resume - Can resume from the last received event

✅ Firewall friendly - Works over standard HTTP/HTTPS ports

✅ Perfect for one-way updates - Server → Client communication

When to use SSE:

Real-time dashboards and monitoring
Live sports scores or stock tickers
News feeds and notifications
Progress indicators for long-running tasks
Social media updates
Server logs streaming

When NOT to use SSE:

Need bidirectional communication (use WebSockets)
Require binary data streaming
Need sub-millisecond latency

Why Ballerina for SSE?

Ballerina is a cloud-native programming language designed for network interactions and integration. Its HTTP module provides first-class support for SSE with a clean, type-safe API. Here's what makes Ballerina special for SSE:

Native stream support - Ballerina's stream type maps perfectly to SSE
Type safety - Compile-time checking for SSE events
Simple syntax - Minimal boilerplate code
Built-in error handling - Graceful error management
Production-ready - Built for enterprise applications

Let's see it in action!

Your First SSE Application in 5 Minutes

Let's build a simple event counter that streams events to clients. This example demonstrates the core SSE pattern in Ballerina.

The Server Code

Here's a complete working SSE server in Ballerina:

import ballerina/http;
import ballerina/lang.runtime;

listener http:Listener counterListener = new(8080);

@http:ServiceConfig {
    cors: {
        allowOrigins: ["*"],
        allowCredentials: false,
        allowHeaders: ["*"],
        allowMethods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"]
    }
}
service /counter on counterListener {

    resource function get events() returns stream<http:SseEvent, error?>|error {
        CounterStream counterStream = new CounterStream();
        stream<http:SseEvent, error?> eventStream = new(counterStream);
        return eventStream;
    }
}

class CounterStream {
    *object:Iterable;
    private int count = 0;
    private final int maxCount = 10;

    public isolated function iterator() returns object {
        public isolated function next() returns record {|http:SseEvent value;|}|error?;
    } {
        return self;
    }

    public isolated function next() returns record {|http:SseEvent value;|}|error? {

        if self.count >= self.maxCount {
            return ();
        }

        self.count += 1;

        http:SseEvent event = {
            data: string `Count: ${self.count}`,
            event: "counter",
            id: self.count.toString()
        };

        runtime:sleep(1);

        return {value: event};
    }
}

That's it! Save this as simple_sse.bal and run:

bal run simple_sse.bal

Testing with curl

curl -N http://localhost:8080/simple/events

Output:

event: counter
id: 1
data: Count: 1

event: counter
id: 2
data: Count: 2

event: counter
id: 3
data: Count: 3
...

The Client Code

Here's a simple HTML client to visualize the events:

<!DOCTYPE html>
<html>
<head>
    <title>SSE Counter Demo</title>
    <style>
        body { font-family: Arial, sans-serif; padding: 20px; }
        #events { background: #f5f5f5; padding: 20px; border-radius: 5px; }
        .event { padding: 10px; margin: 5px 0; background: white; 
                 border-left: 4px solid #2196F3; }
    </style>
</head>
<body>
    <h1>SSE Counter Demo</h1>
    <div id="status">Disconnected</div>
    <div id="events"></div>

    <script>
        const eventSource = new EventSource('http://localhost:8080/simple/events');
        const eventsDiv = document.getElementById('events');
        const statusDiv = document.getElementById('status');

        eventSource.addEventListener('open', () => {
            statusDiv.textContent = 'Connected ✅';
            statusDiv.style.color = 'green';
        });

        eventSource.addEventListener('counter', (event) => {
            const eventDiv = document.createElement('div');
            eventDiv.className = 'event';
            eventDiv.innerHTML = `
                <strong>Event ID:</strong> ${event.lastEventId}<br>
                <strong>Data:</strong> ${event.data}<br>
                <strong>Time:</strong> ${new Date().toLocaleTimeString()}
            `;
            eventsDiv.insertBefore(eventDiv, eventsDiv.firstChild);
        });

        eventSource.addEventListener('error', () => {
            statusDiv.textContent = 'Disconnected ❌';
            statusDiv.style.color = 'red';
        });
    </script>
</body>
</html>

Open this HTML file in your browser, and watch the events stream in real-time!

Understanding the SSE Pattern in Ballerina

Let's break down the key components:

1. The http:SseEvent Record

Every SSE event in Ballerina is represented by the http:SseEvent record:

type SseEvent record {|
    string data;       // The event payload (REQUIRED)
    string event?;     // Event type name (optional)
    string id?;        // Event identifier (optional)
    int retry?;        // Reconnection time in ms (optional)
    string comment?;   // Comment line (optional)
|};

Example:

http:SseEvent event = {
    data: "Hello from server!",
    event: "greeting",
    id: "1",
    retry: 3000  // Client will retry after 3 seconds
};

2. The Stream Class Pattern

Every SSE stream in Ballerina follows this pattern:

class MyEventStream {
    *object:Iterable;  // Step 1: Implement Iterable

    // Step 2: Implement iterator() method
    public isolated function iterator() returns object {
        public isolated function next() returns record {|http:SseEvent value;|}|error?;
    } {
        return self;
    }

    // Step 3: Implement next() method
    public isolated function next() returns record {|http:SseEvent value;|}|error? {
        // Return nil to end the stream
        if shouldEnd {
            return ();
        }

        // Create and return event
        http:SseEvent event = { /* ... */ };
        return {value: event};
    }
}

Key Points:

*object:Iterable makes your class iterable
iterator() returns self, allowing the class to iterate over itself
next() generates each event; return nil to end the stream
The isolated qualifier ensures thread safety

3. The Resource Function

resource function get events() returns stream<http:SseEvent, error?>|error {
    MyEventStream myStream = new MyEventStream();
    stream<http:SseEvent, error?> eventStream = new(myStream);
    return eventStream;
}

The resource function creates the stream and returns it. Ballerina handles all the SSE protocol details automatically!

Real-World Example: Live Stock Ticker

Let's build something more practical—a real-time stock price ticker. This example demonstrates:

Dynamic data generation
Multiple symbols
URL path parameters
Realistic timing

The Stock Ticker Server

import ballerina/http;
import ballerina/lang.runtime;
import ballerina/random;

type Stock record {|
    string symbol;
    decimal price;
    decimal change;
    string timestamp;
|};

listener http:Listener stockListener = new(9090);

service /stocks on stockListener {

    // Stream stock updates for specified symbols
    // Example: /stocks/stream/AAPL,GOOGL,MSFT
    resource function get stream/[string symbols]() returns stream<http:SseEvent, error?>|error {
        string[] symbolList = re `,`.split(symbols);
        StockPriceStream stockStream = new StockPriceStream(symbolList);
        return new(stockStream);
    }
}

class StockPriceStream {
    *object:Iterable;
    private string[] symbols;
    private int eventCount = 0;
    private final int maxEvents = 50;

    function init(string[] symbols) {
        self.symbols = symbols;
    }

    public isolated function iterator() returns object {
        public isolated function next() returns record {|http:SseEvent value;|}|error?;
    } {
        return self;
    }

    public isolated function next() returns record {|http:SseEvent value;|}|error? {
        if self.eventCount >= self.maxEvents {
            return ();
        }

        // Select random stock
        int randomIndex = check random:createIntInRange(0, self.symbols.length());
        string symbol = self.symbols[randomIndex];

        // Generate realistic price data
        decimal basePrice = <decimal>(50.0 + <float>(check random:createIntInRange(0, 950)));
        decimal change = <decimal>(<float>(check random:createIntInRange(-500, 500)) / 100.0);
        decimal currentPrice = basePrice + change;

        Stock stock = {
            symbol: symbol.toUpperAscii(),
            price: currentPrice,
            change: change,
            timestamp: getCurrentTimestamp()
        };

        http:SseEvent event = {
            data: stock.toJsonString(),
            event: "stock-update",
            id: self.eventCount.toString()
        };

        self.eventCount += 1;
        runtime:sleep(1);

        return {value: event};
    }
}

function getCurrentTimestamp() returns string {
    // Simplified - in production, use time:utcNow()
    return "2025-10-30T19:00:00Z";
}

The Stock Ticker Client

<!DOCTYPE html>
<html>
<head>
    <title>Live Stock Ticker</title>
    <style>
        body {
            font-family: Arial, sans-serif;
            background: linear-gradient(135deg, #1e3c72 0%, #2a5298 100%);
            padding: 20px;
        }
        .container {
            max-width: 1200px;
            margin: 0 auto;
            background: white;
            border-radius: 10px;
            padding: 30px;
        }
        .stock-grid {
            display: grid;
            grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
            gap: 20px;
            margin-top: 20px;
        }
        .stock-card {
            padding: 20px;
            border: 2px solid #e0e0e0;
            border-radius: 10px;
            transition: all 0.3s;
        }
        .stock-card.updated {
            animation: pulse 0.5s;
            border-color: #2196F3;
        }
        @keyframes pulse {
            0%, 100% { transform: scale(1); }
            50% { transform: scale(1.05); }
        }
        .symbol {
            font-size: 24px;
            font-weight: bold;
            color: #333;
        }
        .price {
            font-size: 36px;
            font-weight: bold;
            margin: 10px 0;
        }
        .change {
            padding: 5px 10px;
            border-radius: 5px;
            font-weight: bold;
        }
        .positive {
            background: #d4edda;
            color: #155724;
        }
        .negative {
            background: #f8d7da;
            color: #721c24;
        }
    </style>
</head>
<body>
    <div class="container">
        <h1>📈 Live Stock Ticker</h1>
        <div>
            <input type="text" id="symbols" placeholder="AAPL,GOOGL,MSFT,TSLA" 
                   value="AAPL,GOOGL,MSFT,TSLA">
            <button onclick="connect()">Connect</button>
            <button onclick="disconnect()">Disconnect</button>
        </div>
        <div id="status">Disconnected</div>
        <div class="stock-grid" id="stockGrid"></div>
    </div>

    <script>
        let eventSource = null;
        const stockData = new Map();

        function connect() {
            const symbols = document.getElementById('symbols').value;
            if (eventSource) disconnect();

            eventSource = new EventSource(`http://localhost:9090/stocks/stream/${symbols}`);

            eventSource.addEventListener('open', () => {
                document.getElementById('status').textContent = '✅ Connected';
                document.getElementById('status').style.color = 'green';
            });

            eventSource.addEventListener('stock-update', (event) => {
                const stock = JSON.parse(event.data);
                updateStock(stock);
            });

            eventSource.addEventListener('error', () => {
                document.getElementById('status').textContent = '❌ Disconnected';
                document.getElementById('status').style.color = 'red';
            });
        }

        function disconnect() {
            if (eventSource) {
                eventSource.close();
                eventSource = null;
            }
        }

        function updateStock(stock) {
            stockData.set(stock.symbol, stock);
            renderStocks();
        }

        function renderStocks() {
            const grid = document.getElementById('stockGrid');
            grid.innerHTML = '';

            stockData.forEach((stock, symbol) => {
                const card = document.createElement('div');
                card.className = 'stock-card updated';

                const changeClass = stock.change >= 0 ? 'positive' : 'negative';
                const changeSign = stock.change >= 0 ? '+' : '';

                card.innerHTML = `
                    <div class="symbol">${stock.symbol}</div>
                    <div class="price">$${parseFloat(stock.price).toFixed(2)}</div>
                    <div class="change ${changeClass}">
                        ${changeSign}${parseFloat(stock.change).toFixed(2)}
                    </div>
                `;

                grid.appendChild(card);

                setTimeout(() => card.classList.remove('updated'), 500);
            });
        }
    </script>
</body>
</html>

Best Practices for SSE in Ballerina

1. Always Handle Stream Termination

public isolated function next() returns record {|http:SseEvent value;|}|error? {
    // Always provide a way to end the stream
    if self.count >= self.maxCount {
        return (); // End stream gracefully
    }
    // ... generate event
}

2. Use Meaningful Event Types

// Good - descriptive event types
event: "stock-update"
event: "news-article"
event: "user-notification"

// Bad - generic names
event: "data"
event: "msg"

3. Always Send JSON for Complex Data

// Convert records to JSON strings
type StockData record {| string symbol; decimal price; |};
StockData stock = {symbol: "AAPL", price: 150.00};

http:SseEvent event = {
    data: stock.toJsonString(),  // ← Use toJsonString()
    event: "stock-update"
};

4. Implement Proper Error Handling

public isolated function next() returns record {|http:SseEvent value;|}|error? {
    // Use check for operations that can fail
    int randomValue = check random:createIntInRange(0, 100);

    // Or use try-catch for more control
    do {
        int value = check riskyOperation();
        return {value: createEvent(value)};
    } on fail error e {
        // Log error and return safe event
        log:printError("Error generating event", e);
        return {value: createErrorEvent()};
    }
}

5. Control Event Frequency

import ballerina/lang.runtime;

public isolated function next() returns record {|http:SseEvent value;|}|error? {
    // Add delay between events to prevent overwhelming clients
    runtime:sleep(1);  // Wait 1 second

    // Generate and return event
    return {value: event};
}

6. Use Event IDs for Resumption

http:SseEvent event = {
    data: "...",
    id: self.eventCount.toString()  // Client can resume from this ID
};

Clients can use the Last-Event-ID header to resume from where they left off:

const lastEventId = localStorage.getItem('lastEventId');
const eventSource = new EventSource('/events', {
    headers: { 'Last-Event-ID': lastEventId }
});

eventSource.addEventListener('message', (event) => {
    localStorage.setItem('lastEventId', event.lastEventId);
});

7. Set Appropriate Retry Times

http:SseEvent event = {
    data: "...",
    retry: 5000  // Client waits 5 seconds before reconnecting
};

8. Provide Stream Status Events

// Send periodic heartbeat
if self.eventCount % 10 == 0 {
    return {value: {event: "heartbeat", data: "alive"}};
}

// Send stream completion event
if self.count == self.maxCount {
    return {value: {event: "stream-end", data: "completed"}};
}

Performance Considerations

Memory Management

class EfficientStream {
    *object:Iterable;
    private final int maxEvents = 100;  // Limit total events

    public isolated function next() returns record {|http:SseEvent value;|}|error? {
        // Clean up resources periodically
        if self.eventCount >= self.maxEvents {
            self.cleanup();
            return ();
        }
        // ... generate event
    }

    function cleanup() {
        // Release resources
    }
}

Connection Limits

Monitor active SSE connections and implement limits:

// In a real application, track active connections
isolated int activeConnections = 0;
final int MAX_CONNECTIONS = 1000;

resource function get events() returns stream<http:SseEvent, error?>|error {
    lock {
        if activeConnections >= MAX_CONNECTIONS {
            return error("Too many active connections");
        }
        activeConnections += 1;
    }

    // Return stream...
}

Comparison: SSE vs WebSockets vs HTTP Polling

Feature	SSE	WebSockets	HTTP Polling
Connection	One-way (Server → Client)	Two-way	Request-Response
Protocol	HTTP	WebSocket (ws://)	HTTP
Complexity	Low	Medium	Low
Auto-Reconnect	Yes (built-in)	Manual	N/A
Event IDs	Yes	Manual	No
Browser Support	Excellent	Excellent	Universal
Firewall Friendly	Yes	Sometimes blocked	Yes
Use Case	Real-time updates	Chat, gaming	Simple updates
Efficiency	High	Highest	Low

Resources:

Ballerina HTTP Docs: https://lib.ballerina.io/ballerina/http/latest
SSE Specification: https://html.spec.whatwg.org/multipage/server-sent-events.html
MDN SSE Guide: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

Happy Streaming! 🚀

If you found this helpful, share it with your team and let us know what you build with SSE in Ballerina!

Questions or feedback? Drop a comment below or reach out to the Ballerina community.

Advanced Prompt Engineering for goose: A Comprehensive Guide

Abhi — Thu, 30 Oct 2025 08:17:31 +0000

Introduction

While basic prompting can get you started with goose, advanced prompt engineering techniques can dramatically improve the quality, accuracy, and efficiency of your AI-powered development sessions. goose is an open-source AI agent framework developed by Block that automates development tasks, from debugging to deployment, and mastering sophisticated prompting strategies will help you unlock its full potential.

This guide explores expert-level techniques for maximizing goose capabilities through strategic prompt design, context management, and workflow optimization.

Understanding goose architecture

Before diving into advanced techniques, it's crucial to understand how goose works. goose operates through an interactive loop where it receives user input, passes it to an LLM for processing, executes tool calls based on the model's response, and manages context revision to optimize token usage.

Key architectural components:

Interface: Desktop app or CLI where you interact with goose
Agent: Core logic that manages the interactive loop
Extensions: Tools that enable specific capabilities (file management, shell commands, etc.)
Context Management: Automatic revision to keep relevant information and manage token costs

Refer goose architecture for more information.

Core Principles of Advanced Prompting

1. Agentic Thinking: Task-Oriented vs. Chat-Oriented

goose is agentic, not conversational. Understanding this distinction is fundamental:

❌ Conversational approach:

"Can you help me with my tests?"

✅ Agentic approach:

goose, add comprehensive test coverage for src/services/paymentService.js

**Current coverage:** 45% (unacceptable)
**Target coverage:** 90%+

**Testing requirements:**
1. Unit tests for all public methods
2. Integration tests for payment gateway interactions
3. Edge cases and error scenarios
4. Mock external API calls (Stripe)
5. Test success and failure paths
6. Validate input handling
7. Check error message clarity

**Test structure to follow:**
- Use Jest as testing framework
- Follow AAA pattern (Arrange, Act, Assert)
- Group tests by method using describe blocks
- Use descriptive test names (it should...)
- Mock external dependencies properly
- Test one thing per test case

**Specific scenarios to cover:**
- Valid payment processing
- Invalid payment details
- Network failures
- Timeout scenarios
- Insufficient funds
- Currency validation
- Amount validation (positive, not zero, within limits)
- Idempotency (duplicate payment handling)

Key differences:

State the task clearly upfront
Provide measurable success criteria
Include structural requirements
Specify testing/validation approaches
List edge cases explicitly

2. Context Architecture: The .goosehints Strategy

.goosehints is a text file that provides additional context about your project to improve communication with Goose. Advanced users leverage hierarchical hints files strategically.

Hierarchical Context Strategy

All .goosehints files from your current directory up to the root directory are automatically loaded and combined. Structure them by scope:

my-project/
├── .git/
├── .goosehints                 # Project-wide standards
├── frontend/
│   ├── .goosehints            # Frontend-specific hints
│   └── components/
│       ├── .goosehints        # Component-specific hints
│       └── Button.tsx
└── backend/
    ├── .goosehints            # Backend-specific hints
    └── api/
        └── routes.py

Root-level .goosehints example:

# Project Standards

## Code Style
- Use TypeScript strict mode
- Follow Airbnb style guide
- Maximum function length: 50 lines
- Prefer functional components

## Testing Philosophy
- Minimum 80% coverage for new code
- Integration tests for all API endpoints
- Unit tests for business logic
- E2E tests for critical user flows

## Documentation Requirements
- JSDoc comments for all public functions
- README updates for new features
- API documentation in OpenAPI format

## Architecture Patterns
- Use repository pattern for data access
- Implement CQRS for complex domains
- Event-driven architecture for async operations

Component-level .goosehints example:

# Component Guidelines

## Button Component Patterns
- All buttons must support loading state
- Use semantic HTML (<button> not <div>)
- Include aria-label for icon-only buttons
- Follow design system color tokens

## Testing Requirements
- Snapshot tests for visual regression
- Interaction tests with Testing Library
- Accessibility tests with jest-axe

Advanced .goosehints Techniques

1. @-mentions for Dynamic Context Loading

For frequently-needed documentation, use @filename.md or @relative/path/testing.md to automatically include file content in context.

# API Development

When working with API endpoints, reference:
@docs/api-standards.md
@docs/authentication-flow.md

For database operations, see:
@backend/database/schema.sql
@backend/database/migration-guide.md

2. Environment-Specific Hints

# Development Environment

Current environment: ${ENVIRONMENT}

{% if ENVIRONMENT == "production" %}
⚠️ PRODUCTION ENVIRONMENT
- All changes require review
- Database migrations must be reversible
- Feature flags required for new features
{% else %}
Development mode active
- Auto-formatting enabled
- Verbose logging enabled
{% endif %}

3. Custom Command Definitions

You can define useful tasks in .goosehints that benefit from language model-based agent execution flow, like idea generation, summarization, or structured data extraction.

# Custom Commands

## /review
Perform comprehensive code review:
1. Check for security vulnerabilities
2. Verify test coverage
3. Validate documentation
4. Assess performance implications
5. Check accessibility compliance

## /optimize
Analyze and optimize:
1. Identify performance bottlenecks
2. Suggest caching strategies
3. Recommend database query improvements
4. Propose code splitting opportunities

## /summarize [timeframe]
Generate summary of changes:
- Scan git history for [timeframe]
- Group changes by category
- Highlight breaking changes
- Create changelog format
- Save to docs/changelog/YYYY-MM.md

3. Context vs. Memory: Strategic Information Management

Every line in your .goosehints file gets sent with every request to goose, consuming input tokens. This is a critical consideration for advanced users.

Use .goosehints for:

Coding standards and conventions
Project architecture patterns
Testing requirements
Documentation standards
Technology stack information
File structure guidelines

Use Memory Extension for:

User-specific preferences
Dynamic project state
Learned patterns from past interactions
Frequently accessed but changing information
Personal coding habits
Context that needs tags/keywords for retrieval

Token Optimization Strategy:

# Minimal .goosehints (Optimized)

Stack: React 18, TypeScript 5, Node 20
Style: Airbnb, Prettier
Tests: Jest + RTL, 80% coverage
Docs: JSDoc + OpenAPI

[Store detailed standards in Memory Extension]
Use tags: #testing-patterns, #api-conventions, #component-structure

Instead of storing 200+ lines of detailed conventions in .goosehints (processed every request), store them in Memory Extension and retrieve on-demand with tags.

Advanced Prompting Patterns

Pattern 1: Structured Decomposition

Break complex tasks into explicit phases with validation gates.

Refactor the authentication system with the following phased approach:

PHASE 1 - ANALYSIS (Do not proceed to Phase 2 until this is complete)
1. Map current authentication flow
2. Identify all dependencies
3. List potential breaking changes
4. Create rollback strategy
5. Present findings for review

PHASE 2 - TEST COVERAGE (Requires Phase 1 approval)
1. Add tests for existing behavior
2. Achieve 95%+ coverage
3. Document test scenarios
4. Run full test suite
5. Present coverage report

PHASE 3 - IMPLEMENTATION (Requires Phase 2 passing tests)
1. Create feature branch
2. Implement new authentication layer
3. Maintain backward compatibility
4. Update integration tests
5. Run performance benchmarks

PHASE 4 - VALIDATION (Requires Phase 3 success)
1. Execute full test suite
2. Perform security audit
3. Check performance metrics
4. Validate error handling
5. Review documentation

PHASE 5 - DEPLOYMENT (Requires Phase 4 approval)
1. Create deployment plan
2. Set up monitoring
3. Configure feature flags
4. Document rollback procedure
5. Prepare incident response

**Validation Gate**: Pause after each phase and wait for explicit approval before proceeding.

Pattern 2: Perspective-Based Review

Review from multiple perspectives using different expert personas.

Review src/api/payment-processor.ts from multiple expert perspectives:

PERSPECTIVE 1 - SECURITY ANALYST
You are a security expert specializing in payment systems.
Focus on:
- Authentication and authorization
- Data encryption in transit and at rest
- PCI DSS compliance
- Input validation and sanitization
- SQL injection prevention
- XSS vulnerabilities
- Rate limiting
- Audit logging

PERSPECTIVE 2 - PERFORMANCE ENGINEER
You are a performance optimization specialist.
Focus on:
- Database query efficiency
- Caching opportunities
- Memory leaks
- Connection pooling
- Async operation handling
- Load testing considerations
- Scalability bottlenecks

PERSPECTIVE 3 - RELIABILITY ENGINEER
You are an SRE focused on system reliability.
Focus on:
- Error handling and recovery
- Retry logic and backoff strategies
- Circuit breaker patterns
- Timeout configurations
- Monitoring and alerting
- Graceful degradation
- Disaster recovery

PERSPECTIVE 4 - ACCESSIBILITY EXPERT
You are an accessibility specialist.
Focus on:
- Error message clarity
- User feedback mechanisms
- Progressive enhancement
- Screen reader compatibility
- Keyboard navigation
- WCAG compliance

For each perspective:
1. List specific findings (with line numbers)
2. Assess severity (Critical/High/Medium/Low)
3. Provide actionable recommendations
4. Suggest specific code changes

Save complete findings to docs/reviews/payment-processor-review.md

Pattern 3: Constraint-Driven Development

Define explicit constraints that guide implementation decisions.

Implement real-time notification system with these non-negotiable constraints:

PERFORMANCE CONSTRAINTS:
- Latency: < 100ms for notification delivery
- Throughput: Handle 10,000 concurrent connections
- Memory: < 512MB per instance
- CPU: < 40% average utilization

RELIABILITY CONSTRAINTS:
- Uptime: 99.95% SLA
- Message delivery: At-least-once guarantee
- Failover: < 30 seconds recovery time
- Data loss: Zero tolerance

SCALABILITY CONSTRAINTS:
- Horizontal scaling: Auto-scale from 2-20 instances
- Database connections: Pool size max 20
- WebSocket connections: Support 500 per instance
- Queue depth: Max 10,000 messages

SECURITY CONSTRAINTS:
- Authentication: JWT with 15-minute expiry
- Authorization: Role-based access control
- Encryption: TLS 1.3 for all connections
- Rate limiting: 100 requests/minute per user

OPERATIONAL CONSTRAINTS:
- Monitoring: Prometheus metrics exposed
- Logging: Structured JSON logs
- Tracing: OpenTelemetry integration
- Deployment: Zero-downtime deployments

Implementation approach:
1. Choose technology stack that meets ALL constraints
2. Create proof-of-concept demonstrating constraint adherence
3. Implement comprehensive benchmarking
4. Document constraint validation approach
5. Set up automated constraint testing

**Validation**: After implementation, provide evidence that each constraint is met, including benchmark results and test output.

Pattern 4: Error-Driven Refinement

Anticipate and specify error handling comprehensively.

Implement data import pipeline with exhaustive error handling:

ERROR CATEGORIES TO HANDLE:

1. INPUT VALIDATION ERRORS
   - Empty file
   - Wrong file format
   - Corrupted data
   - Encoding issues
   - Schema mismatches
   - Missing required fields
   - Invalid data types
   - Constraint violations

2. PROCESSING ERRORS
   - Timeout during processing
   - Memory overflow
   - Circular dependencies
   - Duplicate detection
   - Data transformation failures
   - Business rule violations

3. EXTERNAL SERVICE ERRORS
   - API rate limit exceeded
   - Service unavailable (503)
   - Authentication failures (401)
   - Authorization failures (403)
   - Network timeouts
   - SSL certificate errors

4. STORAGE ERRORS
   - Database connection lost
   - Transaction rollback needed
   - Constraint violations
   - Deadlock detection
   - Disk space exhausted
   - Write permission denied

For EACH error category:
- Define specific exception types
- Implement retry logic where appropriate
- Set up proper logging with context
- Create user-friendly error messages
- Design recovery strategies
- Add monitoring/alerting
- Write comprehensive tests

Error Response Format:
{
  "status": "error",
  "code": "ERR_VALIDATION_001",
  "message": "User-friendly message",
  "details": "Technical details",
  "timestamp": "ISO-8601",
  "trace_id": "uuid",
  "suggestions": ["Actionable", "recovery", "steps"]
}

Test coverage requirement: 100% for all error paths

Pattern 5: Documentation-First Development

Generate comprehensive documentation before implementation.

Design and document a feature before implementation:

FEATURE: Advanced search with filters

Step 1: Create comprehensive design document at docs/features/advanced-search.md

REQUIRED SECTIONS:

1. OVERVIEW
   - Business objective
   - User story
   - Success metrics
   - Timeline and milestones

2. TECHNICAL DESIGN
   - Architecture diagram
   - Data flow
   - API specifications (OpenAPI)
   - Database schema changes
   - Cache strategy
   - Security considerations

3. USER EXPERIENCE
   - Wireframes (describe in detail)
   - User flows
   - Error states
   - Loading states
   - Edge cases
   - Accessibility requirements

4. IMPLEMENTATION PLAN
   - Task breakdown
   - Dependencies
   - Testing strategy
   - Deployment approach
   - Rollback plan
   - Feature flag configuration

5. ACCEPTANCE CRITERIA
   - Functional requirements (Given/When/Then)
   - Non-functional requirements
   - Browser compatibility
   - Performance benchmarks
   - Security validation

6. RISKS AND MITIGATIONS
   - Technical risks
   - Business risks
   - Mitigation strategies
   - Contingency plans

After documentation is complete and approved:
- Generate test cases from acceptance criteria
- Implement features matching documentation exactly
- Keep documentation updated with any changes
- Create user-facing documentation

Advanced Workflow Patterns

Pattern 6: Iterative Refinement Loop

Build in explicit review cycles.

Implement OAuth2 authentication with iterative refinement:

ITERATION 1 - BASIC IMPLEMENTATION
Implement minimal OAuth2 flow:
- Authorization endpoint
- Token endpoint
- Basic token validation

After implementation, STOP and:
- Run security scan (npm audit, Snyk)
- Execute test suite
- Check performance
- Review code with OWASP guidelines
Present findings before continuing.

ITERATION 2 - ERROR HANDLING
Based on iteration 1 review:
- Add comprehensive error handling
- Implement retry logic
- Add request logging
- Set up monitoring

After implementation, STOP and:
- Test failure scenarios
- Verify error messages
- Check logging output
- Validate monitoring alerts
Present findings before continuing.

ITERATION 3 - SECURITY HARDENING
Based on iteration 2 review:
- Implement rate limiting
- Add CSRF protection
- Set up token rotation
- Configure secure headers

After implementation, STOP and:
- Run penetration tests
- Verify security headers
- Test rate limiting
- Review audit logs
Present findings before continuing.

ITERATION 4 - OPTIMIZATION
Based on iteration 3 review:
- Add caching layer
- Optimize database queries
- Implement connection pooling
- Configure CDN

After implementation:
- Run load tests
- Compare performance metrics
- Document optimizations
Final review and approval.

Pattern 7: Test-Driven Validation

Generate tests that validate requirements explicitly.

Create comprehensive test suite for user registration:

REQUIREMENTS-BASED TEST GENERATION:

For each requirement in docs/requirements/user-registration.md:
1. Generate test case name following pattern: "should [requirement] when [condition]"
2. Write test implementation
3. Add edge case variations
4. Include performance assertions where relevant

REQUIRED TEST CATEGORIES:

1. HAPPY PATH TESTS
   - Valid registration completes successfully
   - User receives confirmation email
   - User can log in with new credentials
   - Profile data is correctly stored

2. VALIDATION TESTS
   For each field (email, password, name, etc.):
   - Empty value rejection
   - Invalid format rejection
   - Boundary value testing
   - SQL injection attempt rejection
   - XSS attempt rejection
   - Unicode character handling

3. BUSINESS LOGIC TESTS
   - Duplicate email rejection
   - Password strength validation
   - Email verification requirement
   - Terms acceptance validation
   - Age verification (if applicable)

4. SECURITY TESTS
   - Password hashing validation
   - Token generation security
   - Session management
   - Rate limiting enforcement
   - CAPTCHA validation

5. INTEGRATION TESTS
   - Email service integration
   - Database transaction handling
   - External API interactions
   - Event publishing verification

6. PERFORMANCE TESTS
   - Registration completes in < 2 seconds
   - Database queries < 100ms
   - Email queuing < 50ms

TEST QUALITY REQUIREMENTS:
- Each test must be independent
- Use factories for test data
- Clean up after each test
- Mock external dependencies
- Include descriptive failure messages
- Assert all observable outcomes
- Achieve 100% branch coverage

Generate tests BEFORE implementation to serve as specification.

Pattern 8: Contextual Problem-Solving

Provide rich context for better solutions.

Debug production issue with full context:

PROBLEM STATEMENT:
Users report intermittent 500 errors during checkout process.

ENVIRONMENTAL CONTEXT:
- Environment: Production
- Region: US-East-1
- Instance count: 12
- Load: ~5,000 requests/minute
- Error rate: 0.5% (started 2 hours ago)
- No recent deployments

TECHNICAL CONTEXT:
- Stack: Node.js 20, Express 4.18, PostgreSQL 14
- Architecture: Microservices with API Gateway
- Cache: Redis 7.0
- Message Queue: RabbitMQ
- Monitoring: DataDog

ERROR DETAILS:
Error: Connection pool exhausted
  at Pool.connect (pg-pool:123)
  at PaymentService.processPayment (payment-service.js:45)
  at OrderController.checkout (order-controller.js:89)

AVAILABLE LOGS:
- Application logs: /var/log/app/*.log
- Database logs: CloudWatch PostgreSQL logs
- API Gateway logs: /var/log/nginx/access.log
- Queue logs: RabbitMQ management console

MONITORING DATA:
- CPU: 45% average (normal)
- Memory: 78% (slightly elevated)
- Database connections: 95/100 (suspicious)
- Cache hit rate: 85% (normal)
- Response time: 250ms avg (normal, except errors)

RECENT CHANGES:
- Marketing campaign launched 3 hours ago
- Traffic increased 30%
- New product category added yesterday

INVESTIGATION APPROACH:
1. Analyze connection pool configuration
2. Check for connection leaks
3. Review recent query patterns
4. Examine transaction handling
5. Check for long-running queries
6. Verify connection timeout settings
7. Review error correlation with traffic spikes

Provide:
- Root cause analysis
- Immediate mitigation steps
- Long-term solution
- Prevention strategy
- Monitoring improvements

Performance Optimization Techniques

Technique 1: Token-Aware Prompting

.goosehints files consume input tokens with every request, which impacts costs for users paying for LLM access. Optimize for token efficiency.

Before (Token-Heavy):

# Detailed Style Guide (1,500+ tokens)

## Function Naming Conventions
Functions should be named using camelCase...
[200 lines of detailed conventions]

## Variable Naming Conventions
Variables should follow these patterns...
[150 lines of detailed patterns]

## Code Organization
Files should be organized in the following manner...
[300 lines of structure details]

After (Token-Optimized):

# Style Guide (Core - 200 tokens)

Style: Airbnb + Prettier
Naming: camelCase functions, PascalCase classes
Structure: feature-based modules
Tests: co-located *.test.ts
Docs: JSDoc public APIs only

[Detailed conventions in Memory Extension]
Tags: #naming, #structure, #testing

Technique 2: Progressive Context Loading

Use @-mentions for frequently-needed documentation to include file content in context, but reference without @ for optional or very large files.

# Project Context (Smart Loading)

## Always Available
@docs/api-core.md              (Essential - auto-load)
@docs/security-requirements.md (Critical - auto-load)

## Load On Demand
Reference docs/api-extended.md      (Use when working on API)
Reference docs/deployment-guide.md  (Use when deploying)
Reference docs/architecture.md      (Use for system design)

## Tags for Memory
Common patterns: #auth-patterns, #db-patterns, #error-handling
Debugging guides: #debug-auth, #debug-database, #debug-api

Technique 3: Session Management

LLMs have context windows, which are limits on how much conversation history they can retain; monitor token usage and start new sessions as needed.

Session Planning Strategy:

# Session 1: Architecture Design
- Design system architecture
- Create API specifications
- Document data models
Total estimated tokens: ~20,000
[Start new session after this]

# Session 2: Implementation (Backend)
- Implement API endpoints
- Write tests
- Set up database
Total estimated tokens: ~30,000
[Start new session after this]

# Session 3: Implementation (Frontend)
- Build UI components
- Integrate with API
- Write E2E tests
Total estimated tokens: ~25,000

# Session 4: Review & Refinement
- Code review
- Performance optimization
- Documentation updates

Advanced Extension Management

Selective Extension Enablement

Turning on too many extensions can degrade performance; enable only essential extensions and tools to improve tool selection accuracy and save context window space.

Strategic Extension Configuration:

# Task-Specific Extension Profiles

## Profile: Backend Development
Extensions:
- Developer (core)
- Database MCP
- Git MCP
- Testing tools
Disabled: Browser, Design tools

## Profile: Frontend Development
Extensions:
- Developer (core)
- Browser Controller
- Component library MCP
- Design system MCP
Disabled: Database tools, Infrastructure

## Profile: DevOps
Extensions:
- Developer (core)
- Docker MCP
- Kubernetes MCP
- Cloud provider MCP
Disabled: UI tools, Testing frameworks

Switch profiles based on task:
`goose configure` → Enable relevant extensions only

Custom MCP Integration

Build custom MCP servers for domain-specific tools.

Create custom MCP server for internal tools:

PURPOSE: Integrate proprietary deployment system

TOOLS TO EXPOSE:
1. deploy_to_staging(service, version, config)
2. promote_to_production(service, version, approvals)
3. rollback_deployment(service, target_version)
4. get_deployment_status(service, environment)
5. trigger_health_check(service, environment)

IMPLEMENTATION REQUIREMENTS:
- OAuth authentication with internal IdP
- Rate limiting (10 requests/minute)
- Comprehensive error handling
- Audit logging for all operations
- Dry-run mode for testing
- Approval workflow integration

INTEGRATION WITH goose:
- Add to .goose/config.yaml
- Document in .goosehints
- Create usage examples
- Set up monitoring

Enable goose to handle deployment workflows end-to-end:
"Deploy user-service v2.3.4 to staging, run tests, and promote to production if tests pass"

Security Best Practices

Pattern 9: .gooseignore for Protection

goose can be eager to make changes; you can stop it from changing specific files by creating a .gooseignore file.

# .gooseignore - Critical File Protection

# Secrets and credentials
.env
.env.*
**/secrets/**
**/credentials/**
**/*_key.pem
**/*_secret.*

# Configuration
**/production.config.*
**/prod.*.yaml
kubernetes/prod/**

# Database
**/migrations/*.sql
**/seeds/production/**

# CI/CD
.github/workflows/production.yml
.circleci/config.yml

# Documentation
docs/architecture/decisions/**
CHANGELOG.md
LICENSE

# Generated files
**/dist/**
**/build/**
**/coverage/**

Pattern 10: Permission-Based Workflows

You can customize supervision levels - Auto Mode for full autonomy, or require approval before actions.

Risk-Based Permission Strategy:

High-Risk Operations (Require Approval):
- Database migrations
- Production deployments
- Dependency updates
- Security-related changes
- API contract modifications

Medium-Risk Operations (Notify):
- Test modifications
- Documentation updates
- Configuration changes
- Refactoring

Low-Risk Operations (Auto-Approve):
- Code formatting
- Comment additions
- Variable renaming
- Import organization

Measuring Success

Prompt Quality Metrics

Track improvement over time:

Evaluate prompt effectiveness:

METRIC 1: First-Try Success Rate
- Track: % of tasks completed without iteration
- Target: > 80% for routine tasks
- Measure: Weekly review of session logs

METRIC 2: Token Efficiency
- Track: Average tokens per completed task
- Target: Reduce by 20% over 3 months
- Measure: Monitor LLM API costs

METRIC 3: Context Relevance
- Track: % of .goosehints content used per session
- Target: > 60% utilization
- Measure: Review context window usage

METRIC 4: Error Recovery
- Track: Steps required to recover from errors
- Target: < 3 steps average
- Measure: Count correction cycles

METRIC 5: Task Complexity Handling
- Track: Successfully completed complex tasks
- Target: Increase by 50% over 6 months
- Measure: Define complexity rubric

Conclusion

Advanced prompt engineering for goose requires understanding its agentic nature, strategically managing context, and designing prompts that guide rather than restrict. By implementing these techniques:

Think in tasks, not conversations - Structure prompts as clear, actionable tasks with success criteria
Architect your context - Use hierarchical .goosehints, Memory Extension, and @-mentions strategically
Optimize for tokens - Balance comprehensive context with cost efficiency
Build in validation - Include explicit review gates and multi-perspective analysis
Iterate systematically - Use structured refinement loops for complex implementations
Measure and improve - Track metrics to continuously enhance your prompting strategy

The most effective goose users treat prompting as a systematic engineering discipline, applying the same rigor to AI interaction as they do to software development. Start with these patterns, experiment with variations, and develop your own sophisticated techniques based on your specific workflows.

Resources

Building Semantica: An AI-Powered Academic Search Platform with MindsDB

Abhi — Wed, 29 Oct 2025 10:09:28 +0000

Connecting ideas across the frontiers of knowledge

🎯 The Problem: Information Overload in Academic Research

Picture this: You're a researcher trying to explore cutting-edge developments in quantum computing. You fire up your favorite search engine and type "quantum error correction." What do you get? Thousands of results. Papers from arXiv, patents, preprints from bioRxiv, medRxiv, chemRxiv—all scattered across different platforms, each with its own search interface.

But here's the real challenge:

Keyword-only search fails to capture semantic meaning: Traditional search engines look for exact keyword matches. If you search for "machine learning in healthcare," you might miss groundbreaking papers that use terms like "artificial intelligence in medical diagnostics" or "neural networks for disease prediction."
Fragmented knowledge sources: Academic papers live in silos—arXiv for physics and CS, bioRxiv for biology, medRxiv for medicine, chemRxiv for chemistry, and patents in yet another database. Researchers waste hours searching across multiple platforms.
No contextual understanding: Found an interesting paper? Great! Now you need to read all 50 pages to understand if it's relevant to your research. What if you could just ask the paper questions?
Comparing multiple papers is tedious: Want to understand how three different approaches to the same problem compare? You'll need to read all three papers, take notes, and synthesize the information yourself.

The research community needed something better.

💡 The Solution: Semantica

Enter Semantica—an intelligent academic search and chat platform that revolutionizes how researchers discover and interact with scientific literature.

Semantica solves these problems by:

✅ Semantic Search: Find papers by meaning, not just keywords

✅ Multi-Source Integration: Search across arXiv, bioRxiv, medRxiv, chemRxiv, and patents simultaneously

✅ AI-Powered Chat: Ask questions and get answers directly from papers

✅ Multi-Document Analysis: Compare and analyze up to 4 papers in a single conversation

✅ Hybrid Search: Combine traditional keyword search with semantic understanding

Github Repo - Semantica
Video Demo - Semantica

🏗️ How I Built It: The MindsDB Magic

The secret sauce behind Semantica is MindsDB—an AI platform that transforms databases into AI-powered systems. Here's how we leveraged MindsDB's powerful features:

1. Knowledge Bases with pgvector Integration

At the heart of Semantica lies MindsDB's knowledge base functionality, which seamlessly integrates with PostgreSQL and the pgvector extension.

What I did:

Created a PostgreSQL database with the pgvector extension for efficient vector storage
Connected it to MindsDB using their database integration
Created a knowledge base that automatically generates embeddings using OpenAI's text-embedding-3-small model

# Sample startup code
def setup_knowledge_base():
    """Create knowledge base in MindsDB with automatic embeddings"""

    create_kb_query = f"""
    CREATE KNOWLEDGE_BASE {kb_name}
    USING
        model = {embedding_model},
        storage = {db_name}.{storage_table};
    """

    client.query(create_kb_query)

Why this is powerful:

Automatic embeddings: MindsDB handles the complexity of generating and storing vector embeddings
Metadata filtering: We can filter by publication year, category, source, and more
Hybrid search: MindsDB supports combining semantic similarity with traditional SQL WHERE clauses

2. Semantic Search with Hybrid Capabilities

One of MindsDB's standout features is its hybrid search capability, which we extensively use in Semantica.

The search query looks like this:

SELECT article_id, metadata, relevance 
FROM my_knowledge_base
WHERE content = 'quantum error correction'
  AND hybrid_search = true 
  AND hybrid_search_alpha = 0.7
  AND source = 'arxiv'
  AND published_year = '2024';

Breaking it down:

content = 'query': Performs semantic similarity search
hybrid_search = true: Enables hybrid mode (semantic + keyword)
hybrid_search_alpha: Controls the balance (0.0 = pure keyword, 1.0 = pure semantic)
Additional WHERE clauses: Traditional SQL filtering on metadata

The result? Users can fine-tune their search from pure semantic understanding to traditional keyword matching, getting the best of both worlds!

3. Dynamic AI Agents for Multi-Document Chat

Here's where things get really exciting. When a user selects papers to chat with, Semantica:

Creates individual knowledge bases for each paper

   CREATE KNOWLEDGE_BASE paper_123_kb
   USING model = text-embedding-3-small,
         storage = my_pgvector.pgvector_storage_table;

   INSERT INTO paper_123_kb
   SELECT text, title, authors, abstract
   FROM my_pgvector.paper_raw
   WHERE article_id = '123' AND source = 'arxiv';

Generates a custom AI agent with access to all selected papers

   CREATE AGENT research_assistant
   USING 
       model = 'gpt-4o',
       skills = [],
       knowledge_bases = ['paper_123_kb', 'paper_456_kb', 'paper_789_kb'],
       prompt_template = 'You are a research assistant...';

Enables natural language queries across all papers

   SELECT answer 
   FROM research_assistant 
   WHERE question = 'Compare the methodologies used in these papers';

Why this approach is brilliant:

Each chat session gets a dedicated AI agent
The agent has deep understanding of all selected papers through their knowledge bases
MindsDB handles the RAG (Retrieval-Augmented Generation) pipeline automatically
Responses are grounded in the actual paper content, reducing hallucinations

4. Reranking for Improved Relevance

MindsDB's knowledge base supports reranking, which we leverage to improve search quality:

knowledge_base:
  reranking_model:
    provider: "openai"
    model_name: "gpt-4o"

Reranking takes the initial search results and uses a more powerful model (GPT-4o) to reorder them based on true relevance to the query. This two-stage approach provides:

Fast initial retrieval using vector similarity
High-quality ranking using advanced language understanding

5. Background jobs to keep up

MindsDB's knowledge base supports jobs, which we leverage to improve keep the knowledge base updated with latest papers:

CREATE JOB kb_sync (
   INSERT INTO kv_kb (
      SELECT * FROM my_pgvector.paper_raw
   )
)
EVERY 1 day;

Every paper with the text and metadata is inserted to postgres table my_pgvector.paper_raw from which it's inserted to the knowledge base.

🛠️ Technical Architecture

The Stack

┌─────────────────────────────────────┐
│   Frontend (React + TypeScript)    │
│   - Search interface                │
│   - Chat UI with PDF viewer         │
│   - Filter controls                 │
└─────────────┬───────────────────────┘
              │ REST API
┌─────────────▼───────────────────────┐
│   Backend (FastAPI + Python)        │
│   - /search endpoint                │
│   - /chat/initiate endpoint         │
│   - /chat/completion endpoint       │
└─────────────┬───────────────────────┘
              │ MindsDB SDK
┌─────────────▼───────────────────────┐
│   MindsDB Platform                  │
│   ┌───────────────────────────┐     │
│   │   Knowledge Bases         │     │
│   │   - Main KB (all papers)  │     │
│   │   - Per-paper KBs (chat)  │     │
│   └───────────────────────────┘     │
│   ┌───────────────────────────┐     │
│   │   AI Agents (GPT-4o)      │     │
│   │   - Dynamic generation    │     │
│   │   - Multi-KB access       │     │
│   └───────────────────────────┘     │
└─────────────┬───────────────────────┘
              │ PostgreSQL Protocol
┌─────────────▼───────────────────────┐
│   PostgreSQL + pgvector             │
│   - Vector embeddings               │
│   - Metadata storage                │
│   - Fast similarity search          │
└─────────────────────────────────────┘

Data Flow for Search

User enters query: "machine learning for optics"
Frontend sends request to /api/v1/search with query and filters
Backend constructs MindsDB query:

   SELECT article_id, metadata, relevance
   FROM my_knowledge_base
   WHERE content = 'machine learning for optics'
     AND hybrid_search = true
     AND hybrid_search_alpha = 0.7;

MindsDB processes query:
- Generates query embedding
- Performs vector similarity search in pgvector
- Applies filters
- Reranks results
Results transformed and returned to frontend
User sees ranked papers with relevance scores

Data Flow for Chat

User selects papers (e.g., 3 papers about quantum computing)
Frontend initiates chat via /api/v1/chat/initiate
Backend creates infrastructure:
- Creates 3 individual knowledge bases (one per paper)
- Populates each KB with paper content
- Creates an AI agent with access to all 3 KBs
- Returns agent ID to frontend
User asks question: "What are the key challenges?"
Frontend sends message via /api/v1/chat/completion
Backend queries agent:

   SELECT answer 
   FROM agent_abc123 
   WHERE question = 'What are the key challenges?';

MindsDB agent:
- Retrieves relevant context from all 3 knowledge bases
- Constructs answer using GPT-4o
- Returns grounded response
Answer displayed in chat interface with markdown formatting

🚀 Key Features in Action

Feature 1: Multi-Source Semantic Search

Users can search across five different academic sources simultaneously:

Example Query: "CRISPR applications in gene therapy"

What happens behind the scenes:

Query is embedded using OpenAI's embedding model
MindsDB performs vector similarity search across all sources
Results are filtered by user-selected corpora
Reranking improves result quality
Papers are returned ranked by relevance

The UX:

Search Results:
📄 "CRISPR-Cas9 Applications in Gene Therapy" (bioRxiv, 2023)
📄 "Therapeutic Genome Editing Methods" (patent, 2024)
📄 "Gene Therapy Advances Using CRISPR" (medRxiv, 2023)

Feature 2: Hybrid Search with Alpha Control

Users can slide between semantic and keyword search:

Alpha = 0.0: Pure keyword matching (fast, specific)
Alpha = 0.5: Balanced hybrid search
Alpha = 1.0: Pure semantic search (finds conceptually similar papers)

Real-world example:

Query: "neural networks"
Alpha 0.0: Returns papers with exact phrase "neural networks"
Alpha 1.0: Returns papers about "deep learning," "artificial neural systems," "connectionist models"

Feature 3: AI-Powered Multi-Document Chat

The killer feature! Select up to 4 papers and have a conversation:

Example conversation:

User: What are the main differences in methodology between these papers?

AI Agent: Based on the three papers you selected:

Paper 1 uses a supervised learning approach with labeled datasets...
Paper 2 employs reinforcement learning with reward shaping...
Paper 3 introduces an unsupervised method using contrastive learning...

The key distinction is in their learning paradigms, with Paper 1 requiring...

What makes this powerful:

The AI has read and understood all papers
Answers are grounded in actual paper content
Can compare, contrast, and synthesize information
Cites specific findings from the papers

Feature 4: Live PDF Viewing

While chatting, users can view the actual PDFs in a split-pane interface:

Left: PDF viewer with Google Docs integration
Right: Chat interface
Switch between papers with one click

📊 The Impact of MindsDB Features

Before MindsDB: The Traditional Approach

Building this without MindsDB would require:

# Manually generate embeddings
from openai import OpenAI
client = OpenAI(api_key=api_key)

def get_embedding(text):
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    )
    return response.data[0].embedding

# Manually implement vector search
import pgvector
def search_papers(query, limit=10):
    query_vector = get_embedding(query)
    cursor.execute("""
        SELECT id, metadata, 
               1 - (embedding <=> %s::vector) as similarity
        FROM papers
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (query_vector, query_vector, limit))
    return cursor.fetchall()

# Manually implement RAG
def answer_question(question, paper_ids):
    # 1. Retrieve relevant chunks from papers
    chunks = retrieve_relevant_chunks(question, paper_ids)

    # 2. Construct prompt with context
    context = "\n".join(chunks)
    prompt = f"Context: {context}\n\nQuestion: {question}\nAnswer:"

    # 3. Call GPT-4
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

# Manually implement hybrid search
def hybrid_search(query, alpha=0.5):
    semantic_results = vector_search(query)
    keyword_results = full_text_search(query)
    return combine_results(semantic_results, keyword_results, alpha)

Problems with this approach:

😰 200+ lines of complex code
🐛 Bug-prone embedding management
🔧 Manual vector database optimization
📈 Difficult to scale
⚡ No built-in reranking
🤯 Complex RAG pipeline implementation

After MindsDB: The Modern Approach

-- Create knowledge base (handles embeddings, storage, indexing)
CREATE KNOWLEDGE_BASE papers_kb
USING model = 'text-embedding-3-small',
      storage = postgres_db.papers;

-- Search with hybrid capabilities
SELECT * FROM papers_kb
WHERE content = 'quantum computing'
  AND hybrid_search = true
  AND hybrid_search_alpha = 0.7;

-- Create AI agent for chat
CREATE AGENT research_agent
USING model = 'gpt-4o',
      knowledge_bases = ['paper1_kb', 'paper2_kb'];

-- Get answers
SELECT answer FROM research_agent
WHERE question = 'What are the key findings?';

Benefits:

✅ Simple, declarative SQL syntax
✅ Automatic embedding generation and management
✅ Built-in hybrid search
✅ Managed RAG pipeline
✅ Optimized vector search
✅ Automatic reranking
✅ Scalable infrastructure

Development time saved: Approximately 2-3 weeks of implementation and testing!

📚 Lessons Learned

1. MindsDB Simplifies AI Pipelines

Key Takeaway: What took weeks to build manually takes hours with MindsDB.

The knowledge base abstraction is incredibly powerful:

No need to manage embedding generation
No need to implement vector search algorithms
No need to build RAG pipelines from scratch
Focus on user experience, not infrastructure

2. SQL as an AI Interface is Powerful

Key Takeaway: Developers already know SQL. Why not use it for AI?

-- This is AI magic disguised as familiar SQL:
SELECT answer FROM ai_agent WHERE question = 'Summarize this paper';

The learning curve is minimal, but the possibilities are vast.

3. Hybrid Search is Essential

Key Takeaway: Don't force users to choose between semantic and keyword search.

Users want both:

Semantic for exploratory research
Keyword for specific citations/terms
Hybrid for balanced results

MindsDB makes this trivial with the hybrid_search_alpha parameter.

4. Dynamic Resource Creation Unlocks Flexibility

Key Takeaway: Creating KBs and agents on-the-fly enables powerful features.

Rather than one monolithic agent:

Create focused, specialized agents per session
Tailor knowledge bases to user selection
Scale horizontally (more sessions = more agents)
Clean architecture with clear boundaries

5. Developer Experience Matters

Key Takeaway: Fast setup = more adoption.

Our automated startup script:

Eliminates manual configuration
Reduces errors
Gets new contributors productive immediately
Serves as living documentation

🌟 Real-World Use Cases

Use Case 1: Literature Review

Scenario: PhD student researching quantum error correction.

Workflow:

Search: "quantum error correction near-term devices"
Enable hybrid search (alpha = 0.7) for balanced results
Filter to 2023-2024, arXiv + patents
Select 4 most relevant papers
Initiate chat
Ask: "What are the different approaches to reducing qubit overhead?"
Ask: "Which paper reports the best error rates?"
Ask: "What are the experimental challenges mentioned?"

Outcome: 30-minute conversation replaces hours of reading and note-taking.

Use Case 2: Cross-Domain Research

Scenario: Biomedical engineer exploring AI applications in medicine.

Workflow:

Search: "machine learning medical diagnostics"
Select sources: arXiv, bioRxiv, medRxiv
Hybrid search to catch both ML papers and medical papers
Find papers bridging CS and medicine
Chat with selected papers to understand interdisciplinary approaches

Outcome: Discovers connections between computer science and medical research that single-domain search would miss.

Use Case 3: Patent Analysis

Scenario: R&D team checking novelty of invention.

Workflow:

Search: "graph neural networks semiconductor design"
Enable patents corpus
Filter to recent years (2022-2024)
Review patents in relevant space
Chat: "What techniques are patented in this domain?"
Chat: "Are there any patents specifically covering [our approach]?"

Outcome: Efficient prior art search with AI-assisted analysis.

Use Case 4: Teaching & Learning

Scenario: Professor preparing lecture on CRISPR.

Workflow:

Search: "CRISPR gene therapy clinical trials"
Select foundational papers + recent advances
Chat: "Explain the evolution from bench to bedside"
Chat: "What are the key safety concerns?"
Chat: "Suggest examples for undergraduate vs graduate courses"

Outcome: AI becomes a teaching assistant, helping structure educational content.

🙏 Acknowledgments

This project wouldn't exist without:

MindsDB Team: For building an incredible AI platform and hosting Hacktoberfest
OpenAI: For GPT-4o and embedding models
PostgreSQL & pgvector: For robust vector storage
FastAPI & React: For excellent developer experiences
The Open Source Community: For countless libraries and tools

Special thanks to MindsDB for making AI accessible to developers worldwide!

#MindsDB #Hacktoberfest #AI #SemanticSearch #AcademicResearch #OpenSource #RAG #VectorDatabase #GPT4 #Python #React

Made with ❤️ for MindsDB Hacktoberfest 2025

RAG Pipelines Made Simple: Ballerina for AI-Powered Search

Abhi — Tue, 28 Oct 2025 21:53:26 +0000

Why building production-ready RAG systems shouldn't feel like assembling IKEA furniture in the dark

The RAG Reality Check

Picture this: You've just convinced your team to build an AI-powered documentation search. The plan is simple—take your company's docs, throw them into a vector database, add some LLM magic, and boom: instant answers for your users.

Three weeks later, you're drowning in boilerplate. Your codebase looks like a Frankenstein monster stitched together from five different libraries. There's error handling scattered everywhere (or worse, nowhere). You're manually chunking documents, wrestling with embedding APIs, debugging vector similarity algorithms, and somehow your "simple" RAG pipeline has morphed into 2,000 lines of glue code.

Sound familiar?

This is where Ballerina walks in, takes a look at your mess, and says: "Why are you working so hard?"

What Makes RAG Systems Complex?

Before we dive into the solution, let's break down why RAG pipelines are deceptively complex:

The RAG Dance (and Why It's Tricky)

A typical RAG workflow involves:

Document Loading: Read files from various sources (PDFs, markdown, databases)
Chunking Strategy: Split documents intelligently without losing context
Embedding Generation: Convert text chunks into vectors using an embedding model
Vector Storage: Store embeddings efficiently with metadata
Query Processing: When a user asks a question, embed their query
Similarity Search: Find the most relevant document chunks
Context Augmentation: Combine retrieved chunks with the user's query
LLM Generation: Send the enriched prompt to an LLM for a final answer

In traditional frameworks, each of these steps requires:

Choosing and configuring multiple libraries
Writing integration code between incompatible APIs
Manual error handling at every boundary
Custom observability instrumentation
Performance optimization from scratch

Most developers spend 70% of their time on plumbing and only 30% on actual AI logic.

Enter Ballerina: Network-Native Meets AI-Native

Ballerina was designed with a radical idea: What if distributed systems programming was actually... easy?

Now, with Ballerina's AI module, this philosophy extends to RAG pipelines. Instead of stitching together separate libraries for embeddings, vector stores, chunking, and LLMs, you get a unified, type-safe API that handles the complexity for you.

The "Aha!" Moment

Let's compare building a RAG system in two approaches:

Traditional Approach (Python + Multiple Libraries):

from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.llms import OpenAI
import os

# Configure everything separately
loader = TextLoader("./leave_policy.md")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(api_key=os.getenv("OPENAI_KEY"))
vectorstore = Chroma.from_documents(chunks, embeddings)

llm = OpenAI(api_key=os.getenv("OPENAI_KEY"))

# Now try to make them work together...
query = "How many leave days can I carry forward?"
docs = vectorstore.similarity_search(query, k=10)
context = "\n".join([doc.page_content for doc in docs])
prompt = f"Context: {context}\n\nQuestion: {query}"
answer = llm(prompt)

Ballerina Approach:

import ballerina/ai;
import ballerina/io;

// Initialize the RAG system
final ai:VectorStore vectorStore = check new ai:InMemoryVectorStore();
final ai:EmbeddingProvider embeddingProvider = 
    check ai:getDefaultEmbeddingProvider();
final ai:KnowledgeBase knowledgeBase = 
    new ai:VectorKnowledgeBase(vectorStore, embeddingProvider);
final ai:ModelProvider modelProvider = check ai:getDefaultModelProvider();

public function main() returns error? {
    // Load and ingest documents
    ai:DataLoader loader = check new ai:TextDataLoader("./leave_policy.md");
    ai:Document|ai:Document[] documents = check loader.load();
    check knowledgeBase.ingest(documents);

    // Query and get answer
    string query = "How many leave days can I carry forward?";
    ai:QueryMatch[] matches = check knowledgeBase.retrieve(query, 10);
    ai:Chunk[] context = from ai:QueryMatch match in matches 
                         select match.chunk;

    ai:ChatUserMessage augmentedQuery = ai:augmentUserQuery(context, query);
    ai:ChatAssistantMessage answer = check modelProvider->chat(augmentedQuery);

    io:println("Answer: ", answer.content);
}

Notice what's different:

No manual chunking configuration (handled intelligently by default)
No explicit embedding API calls (abstracted away)
No prompt engineering boilerplate (built-in augmentation)
No manual context concatenation (query expressions handle it)
Clean error handling with check keyword
Everything is type-safe

Building a Real RAG System: Employee Handbook Assistant

Let's build something practical: an AI assistant that answers questions about your company's employee handbook. We'll cover the complete journey from document ingestion to production deployment.

The Setup: Configuration Made Easy

First, let's understand Ballerina's provider pattern:

import ballerina/ai;
import ballerina/io;

// The VectorStore holds your embeddings
final ai:VectorStore vectorStore = check new ai:InMemoryVectorStore();

// EmbeddingProvider converts text to vectors
// Uses your configured provider (OpenAI, Cohere, etc.) via Ballerina VS Code command
final ai:EmbeddingProvider embeddingProvider = 
    check ai:getDefaultEmbeddingProvider();

// KnowledgeBase combines storage + embeddings
final ai:KnowledgeBase knowledgeBase = 
    new ai:VectorKnowledgeBase(vectorStore, embeddingProvider);

// ModelProvider handles LLM chat completions
final ai:ModelProvider modelProvider = check ai:getDefaultModelProvider();

What's happening here?

Ballerina uses a configuration-based approach. Instead of hardcoding API keys and model names, you configure providers through Ballerina's VS Code extension or configuration files. This means:

No secrets in code
Easy switching between providers (OpenAI, Azure, local models)
Environment-specific configurations
Type-safe provider interfaces

Note: This example uses the default embedding provider and model provider implementations. To generate the necessary configuration, open up the VS Code command palette (Ctrl + Shift + P or command + shift + P), and run the Configure default WSO2 Model Provider command to add your configuration to the Config.toml file.

Step 1: Document Ingestion - Simpler Than You Think

The hardest part of building a RAG system is usually getting data in. Ballerina's DataLoader abstractions make this trivial:

public function ingestEmployeeHandbook() returns error? {
    // Load a single document
    ai:DataLoader policyLoader = check new ai:TextDataLoader("./leave_policy.md");
    ai:Document|ai:Document[] policyDocs = check policyLoader.load();
    check knowledgeBase.ingest(policyDocs);
    io:println("✅ Leave policy ingested");

    // Load multiple documents from a directory
    ai:DataLoader benefitsLoader = check new ai:TextDataLoader("./benefits/");
    ai:Document|ai:Document[] benefitsDocs = check benefitsLoader.load();
    check knowledgeBase.ingest(benefitsDocs);
    io:println("✅ Benefits documentation ingested");

    // You can also load PDFs, Word docs, etc.
    ai:DataLoader handbookLoader = check new ai:PDFDataLoader("./handbook.pdf");
    ai:Document|ai:Document[] handbookDocs = check handbookLoader.load();
    check knowledgeBase.ingest(handbookDocs);
    io:println("✅ Employee handbook ingested");
}

The Magic Behind the Scenes:

When you call knowledgeBase.ingest(), Ballerina automatically:

Chunks your documents using intelligent text splitting (respects sentence boundaries, maintains context)
Generates embeddings for each chunk using your configured embedding provider
Stores vectors with metadata in the vector store
Handles errors gracefully with proper propagation

No manual chunking. No explicit embedding calls. Just load and ingest.

Step 2: The Query Pipeline - Where RAG Shines

Now comes the fun part: answering questions with context from your documents.

type RAGResponse record {|
    string answer;
    Source[] sources;
    float averageRelevance;
|};

type Source record {|
    string content;
    float relevance;
|};

function answerQuestion(string query, int topK = 5) returns RAGResponse|error {
    io:println("\n🔍 Query: ", query);

    // Step 1: Retrieve relevant chunks from knowledge base
    ai:QueryMatch[] queryMatches = check knowledgeBase.retrieve(query, topK);

    // Step 2: Extract chunks for context
    ai:Chunk[] context = from ai:QueryMatch queryMatch in queryMatches
                         select queryMatch.chunk;

    // Step 3: Augment user query with retrieved context
    ai:ChatUserMessage augmentedQuery = ai:augmentUserQuery(context, query);

    // Step 4: Get answer from LLM
    ai:ChatAssistantMessage assistantMessage = 
        check modelProvider->chat(augmentedQuery);

    // Step 5: Calculate average relevance for confidence scoring
    float totalRelevance = 0.0;
    foreach ai:QueryMatch match in queryMatches {
        totalRelevance += match.score;
    }
    float avgRelevance = queryMatches.length() > 0 ? 
        totalRelevance / queryMatches.length() : 0.0;

    // Step 6: Build response with sources
    Source[] sources = from ai:QueryMatch match in queryMatches
                       select {
                           content: match.chunk.content,
                           relevance: match.score
                       };

    return {
        answer: assistantMessage.content,
        sources: sources,
        averageRelevance: avgRelevance
    };
}

Breaking Down the Query Expression:

One of Ballerina's superpowers is query expressions. Look at this beauty:

ai:Chunk[] context = from ai:QueryMatch queryMatch in queryMatches
                     select queryMatch.chunk;

This is not just syntactic sugar. It's a type-safe, functional way to transform data that:

Reads like SQL
Compiles to efficient code
Maintains type safety throughout the pipeline
Makes data transformations explicit and readable

Compare this to Python's list comprehension or JavaScript's map—it's clearer and more maintainable.

Step 3: Understanding Query Augmentation

The ai:augmentUserQuery() function is doing heavy lifting behind the scenes:

ai:ChatUserMessage augmentedQuery = ai:augmentUserQuery(context, query);

This function takes your retrieved chunks and combines them with the user's query using a well-designed prompt template. It's essentially doing:

You are a helpful assistant. Answer the question based on the following context.

Context:
[Chunk 1 content]
[Chunk 2 content]
[Chunk 3 content]

Question: {user's query}

Answer:

But you don't have to worry about prompt engineering—Ballerina handles it with best practices baked in.

Step 4: Making it Production-Ready with HTTP Service

Let's expose our RAG system as a REST API:

import ballerina/http;
import ballerina/ai;
import ballerina/io;

// Initialize RAG components (same as before)
final ai:VectorStore vectorStore = check new ai:InMemoryVectorStore();
final ai:EmbeddingProvider embeddingProvider = 
    check ai:getDefaultEmbeddingProvider();
final ai:KnowledgeBase knowledgeBase = 
    new ai:VectorKnowledgeBase(vectorStore, embeddingProvider);
final ai:ModelProvider modelProvider = check ai:getDefaultModelProvider();

type QueryRequest record {|
    string question;
    int topK = 5;
|};

type QueryResponse record {|
    string answer;
    Source[] sources;
    float confidence;
|};

type Source record {|
    string content;
    float relevance;
|};

service /api on new http:Listener(8080) {

    // Initialize knowledge base on service startup
    function init() returns error? {
        io:println("🚀 Starting Employee Handbook Assistant...");

        // Ingest documents
        ai:DataLoader loader = check new ai:TextDataLoader("./documents/");
        ai:Document|ai:Document[] documents = check loader.load();
        check knowledgeBase.ingest(documents);

        io:println("✅ Knowledge base initialized");
    }

    // Query endpoint
    resource function post query(@http:Payload QueryRequest request) 
            returns QueryResponse|http:InternalServerError {

        // Retrieve relevant chunks
        ai:QueryMatch[]|error queryMatches = 
            knowledgeBase.retrieve(request.question, request.topK);

        if queryMatches is error {
            return {
                body: {
                    message: "Failed to retrieve context",
                    error: queryMatches.message()
                }
            };
        }

        // Extract context chunks
        ai:Chunk[] context = from ai:QueryMatch match in queryMatches
                             select match.chunk;

        // Augment query and get answer
        ai:ChatUserMessage augmentedQuery = 
            ai:augmentUserQuery(context, request.question);
        ai:ChatAssistantMessage|error assistantMessage = 
            modelProvider->chat(augmentedQuery);

        if assistantMessage is error {
            return {
                body: {
                    message: "Failed to generate answer",
                    error: assistantMessage.message()
                }
            };
        }

        // Calculate confidence
        float totalRelevance = 0.0;
        foreach ai:QueryMatch match in queryMatches {
            totalRelevance += match.score;
        }
        float confidence = queryMatches.length() > 0 ? 
            totalRelevance / queryMatches.length() : 0.0;

        // Build sources
        Source[] sources = from ai:QueryMatch match in queryMatches
                           select {
                               content: match.chunk.content,
                               relevance: match.score
                           };

        return {
            answer: assistantMessage.content,
            sources: sources,
            confidence: confidence
        };
    }

    // Health check endpoint
    resource function get health() returns json {
        return {
            status: "healthy",
            service: "Employee Handbook Assistant",
            timestamp: time:utcNow()
        };
    }

    // Add new documents endpoint
    resource function post documents(http:Request req) 
            returns http:Created|http:BadRequest|http:InternalServerError {

        string|http:ClientError filePath = req.getTextPayload();

        if filePath is http:ClientError {
            return {body: "Invalid file path"};
        }

        ai:DataLoader|error loader = new ai:TextDataLoader(filePath);
        if loader is error {
            return {body: "Failed to create data loader"};
        }

        ai:Document|ai:Document[]|error documents = loader.load();
        if documents is error {
            return {body: "Failed to load documents"};
        }

        error? result = knowledgeBase.ingest(documents);
        if result is error {
            return {body: "Failed to ingest documents"};
        }

        return {
            body: {message: "Documents ingested successfully"}
        };
    }
}

What Makes This Special:

Automatic JSON Binding: The @http:Payload annotation automatically deserializes JSON to Ballerina records
Type-Safe Responses: Return types are checked at compile time
Built-in Error Handling: Union types (|error) force you to handle failures
Clean Service Definition: RESTful endpoints with clear resource functions

Learn More

Simplifying Data Persistence in Ballerina: A Deep Dive into `bal persist`

Abhi — Tue, 28 Oct 2025 21:26:11 +0000

Building modern cloud-native applications often requires seamless integration with various data stores. Whether you're working with MySQL, PostgreSQL, Redis, or even Google Sheets, managing data persistence can become complex and time-consuming. Enter bal persist – a powerful feature in the Ballerina programming language that streamlines data persistence across multiple data stores with a unified, type-safe approach.

What is `bal persist`?

bal persist is a comprehensive data persistence feature in Ballerina that allows developers to store and retrieve data across different data stores using a consistent syntax. Think of it as your universal translator for databases – write your code once, and it works seamlessly whether you're using MySQL, MSSQL, PostgreSQL, Redis, Google Sheets, or even in-memory tables.

The beauty of bal persist lies in its simplicity: you don't need to learn different syntaxes for different databases. Define your data model once, and bal persist handles the rest.

The Three Pillars of `bal persist`

The bal persist feature is built on three core components that work together to provide a seamless data persistence experience:

1. Data Model

The data model is the single source of truth for your application's data structure. It's defined using Ballerina record types in a dedicated file within the persist directory of your project.

Here's a simple example of defining an Employee entity:

type Employee record {
    readonly int id;
    string name;
    int age;
    float salary;
}

Key features:

The readonly keyword designates primary key fields
At least one primary key field is required per entity
Multiple primary keys are supported for composite keys
The Ballerina VS Code extension provides validation, code actions, and even Entity Relationship diagram visualization

2. CLI Tool

The bal persist CLI tool is your code generation powerhouse. It automatically generates type-safe client APIs from your data model, along with all necessary configuration files and setup scripts.

Key commands:

persist init: Creates the persist directory and an initial data model file
persist add: Sets up the model file and integrates with bal build for automatic code generation
persist generate: Provides one-time code generation without build integration
persist pull: Introspects existing databases (MySQL, MSSQL, PostgreSQL) to generate data models
persist migrate (experimental): Generates SQL scripts to update database schemas when your data model changes

The best part? With bal build integration, your client APIs are automatically regenerated whenever you build your project!

3. Type-Safe Client API

The generated client API provides a strongly-typed, resource-oriented interface for interacting with your data store. It's generated in the generated directory and includes a Ballerina client object with resources for each entity in your model.

Here's how simple CRUD operations look:

// Create a new employee record
EmployeeInsert employee = {id: 1, name: "John", age: 30, salary: 3000.0};
int[]|error employeeId = sClient->/employees.post([employee]);

// Get an employee record by ID
Employee|error employee = sClient->/employees/1;

// Update an employee record
Employee|error updated = sClient->/employees/1.put({salary: 4000.0});

// Delete an employee record
Employee|error deleted = sClient->/employees/1.delete();

// Get all employees as a stream
stream<Employee, error?> employees = sClient->/employees;

Notice how clean and intuitive the API is – it uses Ballerina's resource access syntax with type safety built in.

How It All Works Together

The bal persist workflow is elegantly simple:

Define Your Data Model: Create entity records in the persist directory using Ballerina record types
Configure Your Data Store: Specify which data store you're using (MySQL, PostgreSQL, Redis, etc.)
Generate Client API: Run bal build or use CLI commands to generate the type-safe client
Use in Your Application: Import and use the generated client in your business logic

Supported Data Stores

bal persist currently supports:

Relational Databases: MySQL, MSSQL, PostgreSQL
In-Memory: In-memory tables for testing and caching
NoSQL: Redis
Cloud Services: Google Sheets

The unified API means switching between data stores is as simple as updating your configuration – no code changes required!

Key Benefits

1. Type Safety

Every operation is type-checked at compile time, reducing runtime errors and improving developer productivity.

2. Database Agnostic

Write your code once and switch between data stores without rewriting your data access layer.

3. Developer Experience

With VS Code integration, you get:

Real-time validation of data models
Code actions for quick fixes
Visual ER diagrams
IntelliSense for generated APIs

4. Automatic Code Generation

No manual boilerplate – the CLI generates everything you need, from client classes to database setup scripts.

5. Migration Support

The experimental persist migrate command helps you evolve your database schema as your application grows.

Real-World Use Cases

Microservices Architecture

Build microservices with different data persistence needs. One service uses PostgreSQL for transactional data, another uses Redis for caching, and a third uses Google Sheets for simple data – all using the same bal persist patterns.

Multi-Tenant Applications

Easily support different data stores for different tenants without code duplication.

Database Migration

Start with an in-memory table for prototyping, then seamlessly migrate to a production database when ready.

Change Data Capture (CDC)

Build CDC services that react to data changes across different data stores with consistent APIs.

Getting Started

Ready to try bal persist? Here's a quick start:

Define your data model in persist/model.bal:

   type User record {
       readonly int id;
       string username;
       string email;
   }

Initialize persist in your Ballerina project:

   bal persist add --datastore mysql

Build your project (this auto-generates the client):

   bal build

Use the generated client in your application:

   import ballerina/io;

   public function main() returns error? {
       Client dbClient = check new();

       UserInsert newUser = {id: 1, username: "alice", email: "alice@example.com"};
       int[] userId = check dbClient->/users.post([newUser]);

       io:println("Created user with ID: ", userId);
   }

Best Practices

Use the VS Code Extension: Leverage the built-in validation and visualization tools
Start Simple: Begin with basic entities and add relationships as needed
Leverage Introspection: Use persist pull to reverse-engineer existing databases
Plan for Migration: Consider using persist migrate (experimental) to manage schema evolution
Test with In-Memory: Use in-memory tables for unit testing before deploying to production databases

The Future of Data Persistence in Ballerina

bal persist represents a significant step forward in how we handle data persistence in cloud-native applications. By providing a unified, type-safe interface across multiple data stores, it eliminates much of the complexity and boilerplate traditionally associated with data access layers.

Whether you're building microservices, REST APIs, or data-intensive applications, bal persist offers a pragmatic approach to data persistence that scales with your needs.

Learn More

Have you tried bal persist in your projects? Share your experiences and use cases in the comments below!

RankGap: Multi-Agent Amazon SEO & Product Visibility Analyzer

Abhi — Sun, 31 Aug 2025 06:49:59 +0000

This is a submission for the AI Agents Challenge powered by n8n and Bright Data

What I Built

Selling on Amazon is one thing; being discovered is another. RankGap is a multi-agent AI system designed to uncover Amazon ranking blind spots for your products using real-time data. It identifies where your product ranks, which search queries you’re missing, and how to optimize for better visibility.

Unlike generic reports, RankGap delivers actionable Amazon SEO insights by combining intelligent search keyword/query generation, ranking analysis, and gap detection.

Why This Matters

Being discovered on Amazon isn’t just luck — it’s about strategic keyword targeting and SEO-aware visibility tracking.

Key challenges sellers face:

Keyword uncertainty: Many sellers don’t know which queries actually drive traffic.
Manual rank tracking is slow: Monitoring dozens of queries across multiple products is tedious.
Visibility gaps cost revenue: Missing even one relevant search query can drastically affect sales.
Real-time complexity: Tracking and analyzing search data dynamically is challenging without automation.

💡 Around 60% of product discoveries on Amazon happen via keyword or query searches.

How it works?

User enters an Amazon product URL (the target product) and selects how many search queries to generate.
The app scrapes the target product details (title, brand, description, features, etc.) using Bright Data.
An LLM generates n search queries based on the product information.
Each query is run on Amazon; the top 20 results are scraped via Bright Data Web Unlocker.
Full product details for those top 20 results are collected using Bright Data’s Amazon scraper.
The target product and its competing results are then analyzed by three specialized LLMs, each receiving only the data it needs.

Demo

Try out the Live App - RankGap

n8n Workflow

There are 3 workflows involved:

Main Workflow
Download Product Data from Amazon
Analysis of the scraped product data

Main Workflow:

The main workflow is the entry point of the entire pipeline. It executes the following steps:

The pipeline is triggered via a webhook (a POST request) with the following payload:

{
    "url": "https://www.amazon.com/dp/B07QTVRF3J", -- amazon product url
    "queryCount": 3, -- number of search queries to generate
    "execution_id": <uuid> -- identifying workflow end result from supabase
}

The given Amazon product is scraped using the Download Product Data from Amazon workflow, returning the full attribute list from Bright Data.
A Search Query Generator agent produces multiple relevant search queries (see AI agents table below for prompts and details).
Search results for each query are scraped using Bright Data Unlocker. The Amazon search URL pattern is:
https://www.amazon.com/s?k=
The HTML returned is parsed, and the top 20 product ASINs are extracted.
The Download Product Data from Amazon workflow scrapes detailed info for these top 20 products.
Data transformations are applied to retain only relevant attributes.
Both the target product and competitors are sent to the Analysis of the Scraped Product Data workflow.
The final analysis is stored in a Supabase database, linked via the execution_id passed at trigger time.

Total number of n8n nodes in this workflow - 19

Workflow json can be found here

Download Product Data from Amazon Workflow:

This workflow’s sole job is to scrape Amazon product data using the Bright Data web scraper. Steps:

Input is a list of product URLs in the following format:

[
    {
        "url": "https://www.amazon.com/dp/B07DSVRF2J"
    },
    {
        "url": "https://www.amazon.com/dp/B07ABCDF2J"
    }
]

Bright Data Verified Node triggers a collection using the Trigger Collection By URL operation, returning a snapshot ID.
Using Monitor Snapshot and an n8n Loop node, the workflow waits until the snapshot is ready.
Once complete, the Download Snapshot operation retrieves product data.

Total number of n8n nodes - 8

Workflow json can be found here

Analysis of the scraped product data Workflow:

This workflow performs structured analysis on the target product and its competitors:

Input: Target product JSON + search results JSON.
Sent to the Visibility & Presence Analysis Agent for ranking visibility.
Sent to the Competitor & Attribute Gap Analysis Agent for feature/attribute gaps.
Sent to the Final Recommendations & Action Plan Agent for actionable next steps.

Workflow json can be found here

Combined total nodes - 19 + 18 + 8 = 45

Technical Implementation

There are 4 AI agents in the complete workflow. Refer the following table for detailed info:

Agent Name	Functionality	LLM Model	BackUp Model	System Prompt Instructions	Memory	Tools
Search Query Generator	Generate natural language keyword and search queries based on input product data	GPT-4o	Gemini 2.5 Pro	https://github.com/Better-Boy/RankGap/blob/main/prompts.txt#L1	NA	Structured Output Parser
Visibility & Presence Analysis Agent	Analyze visibility and presence of the target product in the search results	GPT-4.1-mini	Gemini 2.5 Pro	https://github.com/Better-Boy/RankGap/blob/main/prompts.txt#L20	NA	NA
Competitor & Attribute Gap Analysis	Analyze attribute gap between the target and competitor products	GPT-4.1-mini	Gemini 2.5 Pro	https://github.com/Better-Boy/RankGap/blob/main/prompts.txt#L71	NA	NA
Final Recommendations & Action Plan Agent	Final Recommendation on what can be done better	GPT-4.1-mini	Gemini 2.5 Pro	https://github.com/Better-Boy/RankGap/blob/main/prompts.txt#L122	NA	NA

GPT-4.1-mini was chosen as it has large context windows.

Bright Data Verified Node

The following operations from bright data verified node are used:

Scraping Amazon search page HTML via Web Unlocker
Submitting multiple URLs for product scraping via Trigger Collection by URL
Monitoring snapshot status
Downloading product data snapshots

Application

Tech stack:

Backend - n8n cloud webhook
Database - supabase to store results
Frontend - React + vite + tailwind
Deployment - Vercel

Frontend codebase - https://github.com/Better-Boy/RankGap

Screenshots:

Landing Page & Input:

Results:

Journey

This was my first time working with both n8n and Bright Data. While there was a learning curve, the community support on discord and the documentation made the process smooth.

Key Learnings:

Bright Data’s strength in web data collection became clear when my manual searches kept triggering captchas.
n8n streamlined the entire backend process, saving countless hours of work. It's exhaustive set of nodes is powerful enough to build even the most complex workflows.

Challenges Encountered & How I Overcame Them:

Even when a Bright Data snapshot shows as “ready,” downloading sometimes fails with status: building. Increasing the wait time from 5s → 10s solved this.
In production, cloudflare times out requests after 100 seconds, while n8n workflows can run up to 3 minutes. My workflow often needed 3–5 minutes, so the webhook returned immediately triggering the pipeline as a background process. The pipeline would store the results in supabase with UUID execution_id given by the frontend. By long-polling Supabase from the frontend, I ensured results were fetched once available.

Building RankGap required handling the complexities of Amazon search, real-time scraping, and nested n8n loops — but the result is a powerful system that makes Amazon SEO more transparent and actionable.

If you found this useful, press ❤️ Like or drop a comment.
Thanks for reading! 🚀

PickPerfect - An AI powered recommendation engine

Abhi — Mon, 11 Aug 2025 04:29:22 +0000

This is a submission for the Redis AI Challenge: Real-Time AI Innovators.

What I Built

The Challenge: Milliseconds Matter in E-commerce

Picture this: A user lands on your e-commerce site, browses through a few products, adds something to cart, then hesitates. In those crucial seconds, your recommendation engine needs to:

Process their real-time behavior
Analyze their browsing patterns
Serve personalized recommendations
Do all of this in under 100ms

Traditional recommendation systems using batch processing and SQL databases? They're great for Netflix (where you can wait a few seconds), but in e-commerce, every millisecond of delay costs conversions.

PickPerfect as the name suggests picks the perfect product to recommend to the user based the user's activity like viewing the product page, adding product to cart etc...

Tech Stack

Frontend - vite, react, typescript, tailwind css
Backend - fastapi
Database - Redis
AI engine - RedisAI

Demo

The above image shows geolocation based search for Featured Deals and vector search for Recommended For You

The above image shows product info page.

The above shows vector search on the input query.

The above image shows range based search for numeric fields, exact match for tag fields with OR operator.

The above image shows cart page. Whenever a product is added to the cart, it is tracked.

How I Used Redis 8?

Redis is used in the application almost everywhere. As a real time data layer and as well as permanent storage database. The individual steps where it's is as follows:

Products Indexing

The textual part of the part i.e title, description etc. are concatenated together to form a string. This new string is sent to openAI for embedding which forms the feature vector of the product. This feature vector is indexed in redis full text search document.

redis_client.execute_command(
            f"""
            FT.CREATE {SEARCH_INDEX_NAME} ON JSON PREFIX 1 product: SCHEMA
            $.name AS name TEXT
            $.description AS description TEXT
            $.brand AS brand TAG
            $.price AS price NUMERIC
            $.rating AS rating NUMERIC
            $.reviews AS reviews NUMERIC
            $.category AS category TAG
            $.inStock AS inStock TAG
            $.features[*] AS features TEXT
            $.warehouse_geolocation AS warehouse_location GEO
            $.image AS image TEXT NOINDEX
            $.embedding AS embedding VECTOR FLAT 6 TYPE FLOAT32 DIM {EMBED_DIM} DISTANCE_METRIC {VECTOR_ALGORITHM}
            """
        )

Full Text Search

Products are searched via search filters based on the attributes of the products.

def multi_parameter_search(
        self,
        text_query: Optional[str] = None,
        price_min: Optional[float] = None,
        price_max: Optional[float] = None,
        category: Optional[str] = None,
        brand: Optional[str] = None,
        rating: Optional[float] = None,
        in_stock: Optional[bool] = None,
        geo_location: Optional[Tuple[float, float]] = None,
        geo_radius_km: float = 50,
    ) -> List[Product]:
        """Search products with multiple parameters."""
        query_parts = []
        query_params = {}

        if text_query:
            query_parts.append(f"(@name|@description|@brand|@features:{text_query})")

        if price_min is not None or price_max is not None:
            min_val = price_min if price_min is not None else "-inf"
            max_val = price_max if price_max is not None else "+inf"
            query_parts.append(f"@price:[{min_val} {max_val}]")

        if category:
            query_parts.append(f"@category:{category}")

        if brand:
            query_parts.append(f"@brand:{brand}")

        if rating is not None:
            query_parts.append(f"@rating:[{rating} 5]")

        if in_stock is not None:
            query_parts.append(f"@inStock:{str(in_stock).lower()}")

        if geo_location and geo_radius_km:
            lon, lat = geo_location
            query_parts.append(f"@warehouse_location:[{lon} {lat} {geo_radius_km} km]")

        base_query = " ".join(query_parts) if query_parts else "*"

        query = Query(base_query).return_fields("id").dialect(2)
        query = query.paging(0, 20)

        try:
            result = self.redis_client.ft(settings.search_index_name).search(
                query, query_params=query_params
            )
            filtered_products = []
            for doc in result.docs:
                data = self.redis_client.json().get(getattr(doc, "id"))
                if "embedding" in data:
                    del data["embedding"]
                filtered_products.append(Product(**data))
            return filtered_products
        except Exception as e:
            print(f"Multi-parameter search error: {e}")
            return []

Vector search

Search queries are converted to embedding using openAI and top K nearest neighbors are searched using redis full text search.

def vector_search_embed(self, embedding: List[float], k: int = 10) -> List[Product]:
        """Search products using vector similarity."""
        vector_bytes = np.array(embedding).astype(np.float32).tobytes()
        base_query = f"*=>[KNN {k} @embedding $vec AS vector_score]"
        query = (
            Query(base_query)
            .return_fields("id", "vector_score")
            .sort_by("vector_score")
            .paging(0, k)
        )

        params_dict = {"vec": vector_bytes}

        results = self.redis_client.ft(settings.search_index_name).search(
            query, query_params=params_dict
        )

        filtered_products = []
        for doc in results.docs:
            data = self.redis_client.json().get(getattr(doc, "id"))
            if "embedding" in data:
                del data["embedding"]
            filtered_products.append(Product(**data))
        return filtered_products

Semantic Cache

Redis Semantic cache module of langchain uses RedisVL internally for semantically matching keys. The code for langchain semantic cache can be found in langchain docs.

Geolocation based search

Products in warehouses near to the location of the user can be delivered faster and hence giving extra discount or newer deals becomes a crucial part. This is performed using full text search geo datatype. Reusing the code of full text search.

Personalized recommendation

Whenever an user visits the product page or adds a product to the cart, an event is stored at redis with userId, productId, timestamp. During product recommendation process a sliding window of latest 10 events are retrieved and the embedding vectors of the products are averaged out and again search a vector search is performed on the full text data structure. For personalized recommendations, the user has to be logged in.

As user interests keep changing, time decay method is used to give more importance to the latest events than the older ones.

def get_personalized_recommendations(self, user_email: str) -> List[Product]:
        """Get personalized recommendations for a user."""
        user_events = self.redis_client.lrange(f"user_events:{user_email}", 0, 100)

        if not user_events:
            return product_service.get_trending_products(10)

        product_scores = {}

        for event_str in user_events:
            if isinstance(event_str, bytes):
                event_str = event_str.decode("utf-8")
            event_data = json.loads(event_str)
            product_id = event_data["product_id"]
            event_type = event_data["event_type"]
            timestamp = event_data["timestamp"]

            time_decay = self.calculate_time_decay(timestamp)
            base_score = self.event_weights.get(event_type, 1)
            final_score = base_score * time_decay

            product_scores[product_id] = product_scores.get(product_id, 0) + final_score

        if not product_scores:
            return product_service.get_trending_products(10)

        # Create user preference vector
        weighted_vectors = []

        for product_id, score in product_scores.items():
            embedding = self.redis_client.json().get(
                f"product:{product_id}", "$.embedding"
            )
            if embedding:
                embedding = np.array(embedding).astype(np.float32)
                weighted_vectors.append(embedding * score)

        if weighted_vectors:
            user_preference_vector = np.mean(weighted_vectors, axis=0)
            initial_limit = min(10 * 3, 100)
            results = product_service.vector_search_embed(
                user_preference_vector, initial_limit
            )

            recommendations = []
            for res in results:
                if res.id in product_scores:
                    continue
                recommendations.append(res)

            return recommendations[:10]

        return []

Trending products and categories

Trending products and categories are found out as a result of user event tracking. Redis sorted sets are used here to store the products views.

Code - https://github.com/Better-Boy/PickPerfect

PaperSense: Semantic arXiv Search & Chat Built with MindsDB

Abhi — Sat, 28 Jun 2025 21:38:01 +0000

The Problem

As a developer working with AI, I constantly found myself drowning in arXiv papers. The existing search was purely keyword-based, which meant I'd miss relevant papers that used different terminology. Even worse, after finding papers, I’d lose hours parsing dense academic prose just to understand core ideas.

I needed a system that could:

Understand the semantic meaning behind my search queries
Let me "talk" to research papers directly
Actually understand context, not just keywords

The Solution: A Semantic Search + Chatbot System

I decided to build a comprehensive system using MindsDB's knowledge base capabilities. The final product allows researchers to search semantically through research papers and then have natural language conversations with individual papers or groups of papers.

The tech stack

MindsDB: Handles semantic indexing and chat interface using LLMs.
ChromaDB & PGVector: Stores embeddings for fast similarity search.
FastAPI: Powers a RESTful backend to interact with the front end and MindsDB.
Javascript & CSS: Powers the frontend.

High-Level Architecture

Paperspace Features

Search for papers using natural language
Summarize the paper using AI
Generate new innovative future research directions based on a paper
Chat with the paper (using arxiv id or click on one of the semantic search results)

Lessons Learned

What worked well:

MindsDB: Using AI services as SQL queries saves a lot of development time.
Semantic Search: I found relevant papers I would never have discovered with keyword search
Metadata Enrichment: Including authors and categories in search dramatically improved relevance

What Was Harder Than Expected:

Escaping characters: Academic papers are full of mathematical notation that breaks text processing
Context Window Management: Long papers required careful chunking strategies
Rate Limiting: Both arXiv API and OpenAI had strict limits that required careful orchestration. I lost enough openAI credits while playing with long research papers.

Want to try it out or contribute?

🚀 View the GitHub project

🎥 Watch the YouTube demo