Forem: Ezo Saleh

Self-hosting LLMs for Production Systems: Solving the Model Quality Challenge

Ezo Saleh — Mon, 07 Apr 2025 19:52:10 +0000

In our previous post on semantic caching, we explored how orra's Plan Engine intelligently reuses execution plans to reduce costs and improve performance. Today, we're diving into another powerful capability that enhances orra's flexibility and control for production deployments: self-hosted model support.

The ability to run with self-hosted models addresses several critical needs we've heard from orra users building production multi-agent applications:

Data privacy and security: Keeping sensitive information within your infrastructure
Cost predictability: Trading API costs for infrastructure costs with more predictable scaling
Deployment flexibility: Supporting air-gapped environments and custom setups
Performance tuning: Optimizing for specific workloads and hardware configurations

We'll explore how orra's Plan Engine interfaces with self-hosted models, the architectural decisions that make this possible, and provide practical examples for setting up your own self-hosted model infrastructure for orra.

The Challenge: Bridging Orchestration with Self-hosted Models

When we first designed orra's Plan Engine, we optimized for powerful orchestration capabilities using LLMs with strong reasoning abilities. As our user base grew, we encountered a critical challenge:

How can we address users' data privacy concerns without sacrificing the reliable execution plan generation that's central to orra's orchestration capabilities?

Our early testing revealed a daunting reality: most open-source models struggled significantly with the complex reasoning required to generate valid execution plans. The only open-source model that consistently matched cloud API performance was DeepSeek-R1 — a massive model requiring substantial hardware resources to host and run.

This created a difficult dilemma:

Continue relying on cloud provider APIs (reliable but with data privacy concerns)
Recommend DeepSeek-R1 self-hosting (effective but requiring expensive infrastructure and operational complexity)
Support other open-source models (addressing privacy but potentially sacrificing orchestration reliability)

As one user summarized it during our testing: "Is self-hosting DeepSeek-R1 really worth our time? We need data privacy, but not at the cost of months of infrastructure work."

Through extensive experimentation, we discovered that a newer generation of models like Qwen/QwQ-32B could deliver reliable execution plan generation with the right guidance and validation checks. This opened the door to truly practical self-hosting options that didn't require the most expensive hardware setups.

In the end we decided to support both Qwen/QwQ-32B and DeepSeek-R1 for self-hosted setups.

The Solution: Universal Interfaces, Robust Validation and retries

Our solution to reliable self-hosted orchestration involves two key elements: a universal interface standard and robust validation to ensure execution plan quality across models.

OpenAI-compatible API: The Universal Interface

The first part of our solution leverages the OpenAI-compatible API as a universal interface standard. This choice was strategic rather than just convenient:

// From planengine/llm.go
func (l *LLMClient) Generate(ctx context.Context, prompt string) (response string, err error) {
   l.logger.Trace().
   Str("Prompt", prompt).
   Msg("Generating using prompt")

   request := open.ChatCompletionRequest{
    Model: l.llmModel,
    Messages: []open.ChatCompletionMessage{
      {
        Role:    open.ChatMessageRoleUser,
        Content: prompt,
      },
    },
    Temperature: getModelTemperature(l.llmModel),
   }

   resp, err := l.llmClient.CreateChatCompletion(ctx, request)
   if err != nil {
    if strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "authentication") {
      return "", fmt.Errorf("authentication failed: API key may be required or invalid for this endpoint: %w", err)
    }
    return "", fmt.Errorf("LLM completion failed: %w", err)
   }

   // Process response...
   return resp.Choices[0].Message.Content, nil
}

This approach brought several critical benefits:

Ecosystem Compatibility: Most modern model serving solutions (vLLM, TGI, Ollama) now support this standardized interface
Deployment Flexibility: Organizations can switch between hosted and self-hosted options as their needs evolve
Future Proofing: As new models emerge, they can be integrated without architecture changes

Ensuring Reliable Execution Plans Across Models

The bigger challenge was ensuring that self-hosted models could reliably generate correct execution plans. We discovered that even capable models like Qwen/QwQ-32B required additional guidance and validation compared to cloud APIs.

This led us to implement a validation system with progressive retry logic and detailed feedback:

func (p *PlanEngine) attemptRetryablePreparation(
ctx context.Context, 
......) error {
  // Generate execution plan with potential feedback from previous attempts
  callingPlan, cachedEntryID, isCacheHit, err := p.decomposeAction(
    ctx,
    orchestration,
    orchestration.Action.Content,
    actionParams,
    serviceDescriptions,
    retryCauseIfAny,
  )
  // Various validation stages...

  // Validate action parameters are properly included in TaskZero
  if status, err := p.validateTaskZeroParams(callingPlan, actionParams, retryCount); err != nil {
    p.VectorCache.Remove(orchestration.ProjectID, cachedEntryID)
    return PreparationError{
      Status: status,
      Err:    fmt.Errorf("execution plan action parameters validation failed: %w", err),
    }
  }

  // Validate no composite references to task0 which indicates poor reasoning
  if status, err := p.validateNoCompositeTaskZeroRefs(callingPlan, retryCount); err != nil {
        p.VectorCache.Remove(orchestration.ProjectID, cachedEntryID)
        return PreparationError{
          Status: status,
          Err:    fmt.Errorf("execution plan contains invalid composite task0 references: %w", err),
        }
    }

// More validation steps...
}

This strategy to prompt engineering and validation feedback worked in our favour. When a model fails to generate a valid execution plan, instead of simply rejecting it, we:

Analyze exactly what went wrong (missing parameters, invalid references, etc.)
Provide specific guidance back to the model explaining the issue
Allow the model to retry with this additional context

For example, when parameters aren't properly incorporated into the execution plan, our validateTaskZeroParams method generates precise error messages that become feedback for the next attempt:

Validation Errors:
- Task0 input field 'order_id' doesn't match any action parameter
- Action parameter 'orderId' is missing from Task0 input

This feedback loop significantly improved success rates with models like Qwen/QwQ-32B, making self-hosted orchestration practical without requiring massive models like DeepSeek-R1.

Developer Root Cause Analysis Feedback

Going further, instead of just reporting validation failures with generic errors, we built specific diagnostics that help developers understand the issues they're encountering. Sometimes the model is trying too hard to do the right thing - but essentially this is a
developer error.

Let review an example of this might and how we implemented solution to handle it.

Here's a badly generated plan, which after consecutive retries could not be fixed:

{
    "tasks": [
        {
            "id": "task0",
            "input": {
                "query": "I need a used laptop for college that is powerful enough for programming",
                "budget": "800",
                "category": "laptop"
            }
        },
        {
            "id": "task1",
            "service": "s_aodlipsfcgzmnirkyhyrlmdavwag",
            "input": {
                "query": "$task0.query budget: $task0.budget category: $task0.category"
            }
        }
    ],
    "parallel_groups": [
        [
            "task1"
        ]
    ]
}

The budget and category are extra action parameters that the model is trying to do the right thing by. It doesn't ignore them, instead its injecting them into odd contexts inside task1.input.query creating an invalid input entry.

Larger reasoning models handle this elegantly but smaller ones not so much.

But, actually this is a developer error, budget and category are extra action parameters not required by this job.

The Plan Engine knows this because the service selected to complete the work does not require the params as input. So the final rejection error flags this:

// Final execution plan comprehensive error with root cause analysis
// ORCHESTRATION ERROR: Found invalid task0 references in downstream tasks.
// PROBLEM: The generated execution plan has parameters with invalid task0 references.
// Each input parameter should reference exactly one task0 field without additional text.
// ...
// ...
// HOW TO FIX:
// 1. If required, update each task's service's input schema to accept these parameters
// 2. OR if these parameters aren't needed, remove them from your orchestration request

This approach to error handling and retry logic is what makes reliable self-hosted orchestration possible with models like Qwen/QwQ-32B. You get data privacy without giving up reliable orchestration.

Semantic Caching Impact

The Plan Engine's semantic caching becomes even more valuable with self-hosted models by:

Reducing computational load on your infrastructure
Minimizing memory pressure on GPU resources
Increasing throughput capacity for concurrent users
Providing consistent performance regardless of model location

Future Directions

The addition of self-hosted model support opens up several exciting possibilities for orra:

1. Local Streaming Capabilities

We're working on streaming support for self-hosted models, which will enable real-time feedback during long-running orchestrations without increased API costs.

2. Custom Model Fine-tuning

With self-hosted models, organisations can potentially fine-tune models for specific domains and tasks, creating specialised orchestration engines for their particular use cases.

3. Hardware-specific Optimizations

As orra evolves, we'll provide more guidance on optimizing deployments for specific hardware configurations, including Apple Silicon, AMD GPUs, and specialized AI accelerators.

Conclusion: Flexibility Without Compromise

The addition of self-hosted model support represents a significant milestone in our evolution as a production-ready orchestration platform. By supporting both cloud APIs and self-hosted models through a consistent interface, we've created a system that offers flexibility without compromising on capability or developer experience.

Be sure to check out orra if you want to build production-ready multi-agent applications that handle complex real-world interactions.

For more information on setting up self-hosted models, check out our documentation on model configuration.

Semantic Caching for LLM Execution Plans: How We Cut Costs by 90%

Ezo Saleh — Mon, 17 Mar 2025 22:23:34 +0000

(originally posted here)

In this technical deep dive, we'll explore how orra's Plan Engine implements an intelligent caching system that goes beyond simple key-value lookups. We'll examine the architecture, design decisions, and technical challenges involved in creating a semantic caching layer for LLM-generated execution plans.

When we first built orra's orchestration system, we quickly encountered a challenge that's common in LLM-powered applications: while LLMs provide powerful dynamic orchestration capabilities, they also introduce significant latency and cost overhead.

As request volumes increased in production environments, we found that many actions were semantically similar or functionally equivalent, yet each one triggered a full, expensive LLM call.

This created both performance bottlenecks and unpredictable scaling costs.

Traditional caching approaches didn't work well for this problem.

Exact string matching was too brittle for natural language inputs, and simple parameter templating couldn't handle the flexibility needed for a general-purpose orchestration system. We needed something more sophisticated.

The core challenge we'll address is how to recognise when different user requests are functionally equivalent despite textual differences, and how to efficiently adapt cached plans to work with new parameters.

This involves solving several technical problems:

How to detect semantic similarity between different action statements
How to efficiently map parameters between actions and execution plans
How to adapt cached plans to work with new parameter values
How to manage the cache reliably in a production environment

The approaches described here aren't just specific to orra – they can inform your own work when building LLM-powered applications.

As LLMs are incorporated into production systems, techniques for semantic caching, dynamic parameter substitution, and vector similarity can be applied to reduce costs, improve response times, and enhance user experiences across many different contexts.

By the end of this article, you'll understand the inner workings of orra's caching system, the tradeoffs involved in its design, and how these principles can be applied to significantly improve both performance and cost-efficiency in your own LLM-powered applications.

Understanding Execution Plans

At the heart of orra's orchestration system is the Plan Engine - responsible for coordinating multi-agent workflows through execution plans. An execution plan is a structured representation of how different services and agents should interact to accomplish a user's requested action.

Here's a simplified example:

{
  "tasks": [
    {
      "id": "task0",
      "input": {
        "customerId": "CUST789",
        "orderId": "ORD456"
      }
    },
    {
      "id": "task1",
      "service": "customer-service",
      "input": {
        "customerId": "$task0.customerId"
      }
    },
    {
      "id": "task2",
      "service": "order-system",
      "input": {
        "orderId": "$task0.orderId"
      }
    }
  ],
  "parallel_groups": [
    ["task1", "task2"]
  ]
}

These plans are generated by the Plan Engine through an LLM, which analyzes the user's intent and creates an optimized coordination structure. While effective, this approach presents two significant challenges:

Latency: Generating execution plans via an LLM introduces latency (often 1-3 seconds), degrading the user experience
Cost: Each plan generation incurs an API call cost, which becomes significant at scale

This is where the Plan Engine's caching system comes in. By recognising when new actions are semantically similar to previously processed ones, it can reuse and adapt existing plans rather than generating new ones each time.

The Core Insight: Semantic Understanding with Parameter Adaptation

Most caching systems work as simple lookup tables - they either find an exact match or they don't.

When building orra's Plan Engine, we asked ourselves:

Could we create a caching system that understands when two differently worded requests are asking for the same thing, and then adapt a cached plan to work with new parameters?

The result is a semantic caching layer that:

Recognises when actions are functionally equivalent, even with different phrasing
Dynamically adapts cached plans to work with new input parameters
Preserves the orchestration structure while updating specific values

This approach dramatically changes the performance and cost profile of LLM-driven orchestration:

Traditional approach:
User1: "Process order #1234" → Full LLM API call (1-3 seconds)
User2: "Process order #5678" → Another full LLM API call (1-3 seconds)

With semantic caching:
User1: "Process order #1234" → Full LLM API call (1-3 seconds) + cache storage
User2: "Process order #5678" → Semantic match + parameter substitution (~50ms, no LLM API call)

For applications handling thousands or millions of similar but distinct requests, this can reduce both latency and API costs by over 90%.

Let's explore how the Plan Engine accomplishes this.

Vector-Based Similarity Matching: Recognizing Intent

The first part of the solution involves determining when actions are semantically similar. orra's Plan Engine uses vector embeddings to represent the meaning of actions rather than just their text.

Implementation Details

func (pc *ProjectCache) findBestMatch(query CacheQuery) (*CacheEntry, float64) {
    // Lock for concurrent access safety
    pc.mu.RLock()
    defer pc.mu.RUnlock()

    var bestScore float64 = -1
    var bestEntry *CacheEntry

    // First pass: quick filtering
    for _, entry := range pc.entries {
        // Filter out entries with different service signatures
        if entry.ServicesHash != query.servicesHash {
            continue
        }

        // Filter based on grounding state
        if entry.Grounded != query.grounded {
            continue
        }

        // Calculate semantic similarity
        score := CosineSimilarity(query.actionVector, entry.ActionVector)

        // Track best match
        if score > bestScore {
            bestScore = score
            bestEntry = entry

            // Early exit for near-perfect matches
            if score > 0.999 {
                break
            }
        }
    }

    return bestEntry, bestScore
}

The Plan Engine converts action text into vector embeddings, creating a mathematical representation of semantic meaning. By computing cosine similarity between these vectors, it can identify when two differently worded requests are functionally equivalent.

Several questions needed to be addressed in this approach:

How do we quickly filter out irrelevant matches? The solution was to first check if the service environment is identical (via hash comparison) before computing expensive vector similarities.
When is a match "close enough"? Through experimentation, we found a threshold of 0.95 similarity provides a good balance between reusability and precision.
How do we handle domain-specific knowledge? Plans using domain grounding are cached separately from non-grounded plans to preserve domain-specific constraints.

Dynamic Parameter Substitution: Adapting Plans to New Inputs

Finding similar plans solves half the challenge. The next question was:

How can we adapt a cached plan to work with different parameter values while maintaining its structure and validity?

Parameter Mapping and Substitution

When a plan is first generated and cached, the Plan Engine analyzes which task inputs correspond to action parameters:

func extractParamMappings(actionParams ActionParams, task0Input map[string]interface{}) ([]TaskZeroCacheMapping, error) {
    // Maps for different value types
    stringValues := make(map[string]string) // For primitive values
    jsonValues := make(map[string]string)   // For complex types

    // Build lookup maps from action parameters
    for _, param := range actionParams {
        field := param.Field

        // For primitive types (strings, numbers, booleans)
        if isPrimitive(param.Value) {
            stringValues[fmt.Sprintf("%v", param.Value)] = field
        } else {
            // For complex types (arrays, objects), use JSON representation
            jsonBytes, err := json.Marshal(param.Value)
            if err == nil {
                jsonValues[string(jsonBytes)] = field
            }
        }
    }

    var mappings []TaskZeroCacheMapping

    // Find Task0 input values that match action param values
    for task0Field, task0Value := range task0Input {
        matched := false
        actionField := ""
        valueToStore := ""

        // Try to match primitive values
        if isPrimitive(task0Value) {
            strVal := fmt.Sprintf("%v", task0Value)
            if field, ok := stringValues[strVal]; ok {
                matched = true
                actionField = field
                valueToStore = strVal
            }
        } else {
            // Try to match complex values via JSON comparison
            jsonBytes, err := json.Marshal(task0Value)
            if err == nil {
                jsonStr := string(jsonBytes)
                if field, ok := jsonValues[jsonStr]; ok {
                    matched = true
                    actionField = field
                    valueToStore = jsonStr
                }
            }
        }

        if matched {
            mappings = append(mappings, TaskZeroCacheMapping{
                Field:       task0Field,
                ActionField: actionField,
                Value:       valueToStore,
            })
        }
    }

    return mappings, nil
}

When a cache hit occurs, the Plan Engine uses these mappings to generate a new version of the plan with updated parameters:

func substituteTask0Params(content string, originalInput, newParams json.RawMessage, mappings []TaskZeroCacheMapping) (string, error) {
    // Parse the execution plan
    var plan ExecutionPlan
    if err := json.Unmarshal([]byte(content), &plan); err != nil {
        return "", fmt.Errorf("failed to parse calling plan for task0 param substitution: %w", err)
    }

    // Parse original Task0 input
    var origTask0Input map[string]interface{}
    if err := json.Unmarshal(originalInput, &origTask0Input); err != nil {
        return "", fmt.Errorf("failed to parse original Task0 input: %w", err)
    }

    // Parse new action params
    var actionParams ActionParams
    if err := json.Unmarshal(newParams, &actionParams); err != nil {
        return "", fmt.Errorf("failed to parse new action params: %w", err)
    }

    // Generate new Task0 input using mappings
    newTask0Input, err := applyParamMappings(origTask0Input, actionParams, mappings)
    if err != nil {
        return "", err
    }

    // Find and update Task0 in the plan
    task0Found := false
    for i, task := range plan.Tasks {
        if task.ID == "task0" {
            plan.Tasks[i].Input = newTask0Input
            task0Found = true
            break
        }
    }

    if !task0Found {
        return "", fmt.Errorf("task0 not found in calling plan")
    }

    // Marshal the updated plan
    updatedContent, err := json.Marshal(plan)
    if err != nil {
        return "", fmt.Errorf("failed to marshal updated plan: %w", err)
    }

    return string(updatedContent), nil
}

Several challenging questions had to be solved in this process:

How do we identify which parameters matter? The system uses value-based matching rather than just field names, enabling more flexible correspondence detection.
How do we handle complex data types? Both primitive values and complex types (arrays, objects) are supported through different comparison mechanisms.
How do we ensure parameter references remain valid? The structure of the plan is preserved, with only the specific parameter values being updated.

Cache Management: Ensuring Reliability

To be trusted in production environments, a caching system needs robust management capabilities. Here are some of the questions we addressed in the Plan Engine's cache management:

Project Isolation

How do we keep one project's cache from affecting others?

Each project maintains its own isolated cache:

func (c *VectorCache) getProjectCache(projectID string) *ProjectCache {
    c.mu.Lock()
    defer c.mu.Unlock()

    pc, exists := c.projectCaches[projectID]
    if !exists {
        pc = newProjectCache(c.logger)
        c.projectCaches[projectID] = pc
        c.logger.Info().
            Str("projectID", projectID).
            Msg("Created new project cache")
    }

    return pc
}

This design ensures:

Separate security boundaries between projects
Independent cache sizing and policies
One project's cache usage doesn't affect others

Time-based Expiration and Size Limits

How do we prevent the cache from growing indefinitely?

The Plan Engine implements both TTL-based expiration and size constraints:

func (c *VectorCache) cache(projectID string, planJson string, actionVector *mat.VecDense, servicesHash string, task0Input json.RawMessage, taskZeroCacheMappings []TaskZeroCacheMapping, actionWithFields string, grounded bool) *CacheEntry {
    pc := c.getProjectCache(projectID)

    // Create new cache entry
    entry := &CacheEntry{
        ID:                     uuid.New().String(),
        Response:               planJson,
        ActionVector:           actionVector,
        ServicesHash:           servicesHash,
        Task0Input:             task0Input,
        CacheMappings:          taskZeroCacheMappings,
        Timestamp:              time.Now(),
        CachedActionWithFields: actionWithFields,
        Grounded:               grounded,
    }

    // Add to project cache with size management
    pc.mu.Lock()
    if len(pc.entries) >= c.maxSize {
        // Remove oldest entry
        pc.entries = pc.entries[1:]
    }
    pc.entries = append(pc.entries, entry)
    pc.mu.Unlock()

    return entry
}

The system also includes automatic cleanup of expired entries:

func (c *VectorCache) cleanup() {
    c.mu.RLock()
    defer c.mu.RUnlock()

    now := time.Now()
    for projectID, pc := range c.projectCaches {
        pc.mu.Lock()
        var validIdx int
        for i, entry := range pc.entries {
            if now.Sub(entry.Timestamp) < c.ttl {
                if validIdx != i {
                    pc.entries[validIdx] = entry
                }
                validIdx++
            }
        }
        pc.entries = pc.entries[:validIdx]
        pc.mu.Unlock()

        c.logger.Debug().
            Str("projectID", projectID).
            Int("remainingEntries", validIdx).
            Msg("Cleaned project cache")
    }
}

Concurrency Protection

How do we handle multiple similar requests arriving simultaneously?

The Plan Engine uses a singleflight pattern to prevent duplicate LLM calls for the same action:

func (c *VectorCache) Get(ctx context.Context, projectID, action string, actionParams json.RawMessage, serviceDescriptions string, groundingHit *GroundingHit, backPromptContext string) (*CacheResult, json.RawMessage, error) {
    result, err, _ := c.group.Do(fmt.Sprintf("%s:%s", projectID, action), func() (interface{}, error) {
        return c.getWithRetry(ctx, projectID, action, actionParams, serviceDescriptions, groundingHit, backPromptContext)
    })

    if err != nil {
        return nil, nil, err
    }

    cacheResult := result.(*CacheResult)
    // ...
}

This ensures that concurrent requests for the same action don't result in duplicate LLM calls, even before the result is cached.

Practical Benefits in Production

The combination of semantic matching and parameter substitution delivers several concrete benefits for developers building multi-agent applications:

Significantly reduced latency: Cache hits respond in milliseconds rather than seconds
Lower operating costs: For applications with repetitive patterns, costs scale with unique patterns rather than request volume
More consistent behavior: Similar requests receive consistently structured plans
Transparent to developers: No explicit cache management or pattern definition required
Integration with domain grounding: Maintains domain-specific knowledge while improving performance

Challenges and Limitations

The Plan Engine's caching system isn't without trade-offs. Some challenges include:

Memory usage: Storing vectors and plan structures requires more memory than simple response caching
Cold start: New deployments need time to build up an effective cache
Service sensitivity: Changes to service definitions invalidate cache entries
Parameter detection edge cases: The current approach might miss some parameter relationships

Conclusion: Lessons Learned and Future Directions

orra's Plan Engine demonstrates a practical approach to improving both the performance and cost-efficiency of LLM-driven applications. By recognizing semantic similarity between actions and dynamically adapting cached plans to new parameters, it provides a path to significantly faster response times and lower API costs, without sacrificing the flexibility that makes LLMs valuable.

Through building this system, we've learned several important lessons about balancing performance, cost, and capability in LLM-driven orchestration. At the same time, we've identified several areas for continued improvement:

Parameter Name Sensitivity

Currently, the system depends on action parameter names being identical between requests. If a developer changes parameter names but keeps the same underlying values and structure, it results in a cache miss. For example:

// Will hit cache if previously seen
action: "Process order", params: { orderId: "12345" }

// Will miss cache, despite functional equivalence
action: "Process order", params: { order_id: "12345" }

We're exploring techniques to recognise functionally equivalent parameter structures despite naming differences, potentially using schema matching or more advanced embedding techniques.

Embedding Model Economics

While using embedding models for semantic matching is effective, it raises questions about cost at scale. We're investigating:

The feasibility of using self-hosted embedding models instead of OpenAI's APIs
The performance/cost tradeoffs of different model sizes and architectures
Potential for batch processing or other optimizations to reduce embedding costs

Initial testing suggests local models can reduce costs but may impact match quality or latency, so finding the right balance requires careful evaluation.

Cache Persistence

The current implementation stores cache entries in memory, meaning the cache is lost if the Plan Engine restarts. This creates a "cold start" problem where performance and cost benefits are temporarily lost after deployments or outages.
We plan to implement cache persistence to disk or a database, allowing the Plan Engine to:

Rebuild its cache state quickly after restarts
Share cache entries across multiple instances
Preserve valuable cache entries during maintenance windows

Finally

Hopefully you've found this useful!

Be sure to check out orra if you want to build production-ready multi-agent applications that handle complex real-world interactions.