Forem: Elizabeth Fuentes L

Los agentes de IA pueden alucinar el éxito de una operación incluso cuando violan reglas de negocio. Confirman reservas sin verificación de pago, aceptan parámetros inválidos como 15 huéspedes cuando el máximo es 10, o ignoran prerrequisitos obligatorios.

El problema central: los agentes pueden confirmar reservas de hotel a pesar de que el pago nunca fue verificado, violar restricciones de capacidad, o saltarse pasos de validación obligatorios. El prompt engineering por sí solo no puede prevenir estos errores.

Este post demuestra cómo la validación neurosimbólica — combinando razonamiento del LLM con reglas simbólicas deterministas aplicadas a nivel de framework — bloquea operaciones inválidas antes de que se ejecuten.

Esta demo utiliza Strands Agents. Patrones similares pueden aplicarse en LangGraph, AutoGen u otros frameworks de agentes.

Resumen de la Serie

Esta es la Parte 3 de una serie sobre cómo detener las hallucinations en agentes de IA:

RAG vs Graph-RAG — Knowledge graphs para prevenir hallucinations
Semantic Tool Selection — Filtrado de tools basado en vectores
AI Agent Guardrails (este post) — Reglas simbólicas que los LLM no pueden evadir
Runtime Guardrails — Redirigir en lugar de bloquear
Multi-Agent Validation — Detección de hallucinations basada en equipos

El Problema: Los Prompts Son Sugerencias, No Restricciones

La investigación de ATA: Autonomous Trustworthy Agents (2024) identifica tres patrones de hallucination:

Errores de parámetros: El agent llama a book_hotel(guests=15) a pesar de que el docstring indica un máximo de 10
Errores de completitud: El agent ejecuta reservas sin la verificación de pago requerida
Comportamiento de evasión de tools: El agent confirma éxito sin llamar a los tools de validación obligatorios

Causa raíz: Los prompts son texto que el LLM interpreta. Las reglas de negocio embebidas en docstrings se convierten en sugerencias — el modelo decide si las sigue en cada llamada.

La Solución: Validación Neurosimbólica con Strands Hooks

Los hooks a nivel de framework interceptan las llamadas a tools antes de la ejecución. Usando BeforeToolCallEvent en Strands Agents, puedes:

Validar reglas simbólicas de forma determinista
Cancelar la ejecución del tool si alguna regla falla
Enviar un mensaje de cancelación directamente al LLM (no puede ser anulado)

Diferencia arquitectónica clave:

Prompts: Entrada del LLM (interpretable, anulable)
Hooks: Interceptores del framework (deterministas, obligatorios)

Prerrequisitos

Experiencia con Python y uso de tools en agentes LLM
Familiaridad con Strands Agents

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/04-neurosymbolic-demo
uv venv && uv pip install -r requirements.txt

Implementación: Reglas, Hook y Dos Agentes

Paso 1: Definir Reglas Simbólicas

from dataclasses import dataclass
from typing import Callable

@dataclass
class Rule:
    name: str
    condition: Callable[[dict], bool]
    message: str

BOOKING_RULES = [
    Rule("max_guests",   lambda ctx: ctx.get("guests", 1) <= 10,     "Maximum 10 guests per booking"),
    Rule("valid_dates",  lambda ctx: ctx["check_in"] < ctx["check_out"], "Check-in must be before check-out"),
]

CONFIRMATION_RULES = [
    Rule("payment_before_confirm", lambda ctx: ctx.get("payment_verified", False),
         "Payment must be verified before confirmation"),
]

Las reglas son funciones Python simples — deterministas, testeables y auditables de forma independiente a cualquier agent.

Paso 2: Crear el Hook de Validación

from strands.hooks import HookProvider, HookRegistry, BeforeToolCallEvent

class NeurosymbolicHook(HookProvider):

    def register_hooks(self, registry: HookRegistry) -> None:
        registry.add_callback(BeforeToolCallEvent, self.validate)

    def validate(self, event: BeforeToolCallEvent) -> None:
        tool_name = event.tool_use["name"]
        if tool_name not in self.rules:
            return
        context = self._build_context(tool_name, event.tool_use["input"])
        passed, violations = validate(self.rules[tool_name], context)
        if not passed:
            event.cancel_tool = f"BLOCKED: {', '.join(violations)}"

Cuando se establece event.cancel_tool, Strands reemplaza el resultado del tool con ese mensaje antes de que el LLM lo vea. El tool nunca se ejecuta.

Paso 3: Definir Tools Limpios

from strands import tool

@tool
def book_hotel(hotel: str, check_in: str, check_out: str, guests: int = 1) -> str:
    """Book a hotel room."""
    return f"SUCCESS: Booked {hotel} for {guests} guests, {check_in} to {check_out}"

@tool
def process_payment(amount: float, booking_id: str) -> str:
    """Process payment for a booking."""
    return f"SUCCESS: Processed ${amount:.2f} for {booking_id}"

@tool
def confirm_booking(booking_id: str) -> str:
    """Confirm a booking."""
    return f"SUCCESS: Confirmed {booking_id}"

Los tools contienen solo lógica de negocio — sin validación mezclada.

Paso 4: Crear Ambos Agentes

from strands import Agent
# Using OpenAI-compatible interface via Strands SDK (not direct OpenAI usage)
from strands.models.openai import OpenAIModel

MODEL = OpenAIModel(model_id="gpt-4o-mini")

# Baseline: no hook, no validation
baseline_agent = Agent(tools=[book_hotel, process_payment, confirm_booking], model=MODEL)

# Guarded: hook intercepts every tool call
hook = NeurosymbolicHook(STATE)
guarded_agent = Agent(tools=[book_hotel, process_payment, confirm_booking], hooks=[hook], model=MODEL)

Mismos tools, mismo modelo, mismos prompts. La única diferencia: hooks=[hook]

Resultados de las Pruebas

Prueba 1: Confirmar Reserva Sin Pago

Consulta: "Confirm booking BK001"

Agent	Resultado
Baseline	❌ Ejecuta — pago nunca verificado, docstring ignorado
Guarded	✅ Bloqueado — el hook evalúa `CONFIRMATION_RULES`, encuentra `payment_verified=False`, cancela

El LLM recibe: BLOCKED: Payment must be verified before confirmation

Prueba 2: Reservar Hotel Excediendo el Límite de Huéspedes

Consulta: "Book Grand Hotel for 15 people from 2026-03-20 to 2026-03-25"

Agent	Resultado
Baseline	❌ Ejecuta con 15 huéspedes — máximo en docstring ignorado
Guarded	✅ Bloqueado — `max_guests_check(15 <= 10)` falla, ejecución cancelada

La validación ocurre antes de la ejecución — no hay reserva que revertir.

Prueba 3: Reserva Válida

Consulta: "Book Grand Hotel for 5 guests from 2026-03-20 to 2026-03-25"

Agent	Resultado
Baseline	✅ Ejecuta
Guarded	✅ Ejecuta — todas las reglas pasan, el hook no agrega fricción

Resumen de Escenarios

Escenario	Baseline	Guarded
Confirmar sin pago	❌ Ejecuta — hallucination	✅ Bloqueado antes de la ejecución
Reservar 15 huéspedes (máx. 10)	❌ Ejecuta — regla violada	✅ Bloqueado antes de la ejecución
Reserva válida (5 huéspedes)	✅ Ejecuta	✅ Ejecuta

Punto Clave: Dónde Se Aplica la Validación

Prompt Engineering (Insuficiente)

system_prompt = """
IMPORTANT: Never confirm bookings without payment verification.
CRITICAL: Maximum 10 guests per booking.
"""
# The LLM reads this as context. It can hallucinate compliance.

❌ El LLM decide si sigue estas instrucciones en cada llamada.

Aplicación Basada en Hooks (Suficiente)

def validate(self, event: BeforeToolCallEvent) -> None:
    passed, violations = validate(self.rules[tool_name], context)
    if not passed:
        event.cancel_tool = f"BLOCKED: {', '.join(violations)}"
        # Tool never executes. LLM receives the cancellation.
        # There is no path to override this.

✅ El hook se ejecuta fuera del LLM. La decisión no depende del LLM.

Diferencia arquitectónica: Los prompts son entrada del LLM (interpretable). Los hooks son interceptores del framework (deterministas).

Consideraciones para Producción

Ventajas

Restricciones verificables: Las reglas son código, no instrucciones
Centralizadas: Un solo hook valida todos los tools
Testeables de forma independiente: Las reglas funcionan fuera de cualquier agent o contexto de LLM
Auditables: Las violaciones de reglas producen eventos explícitos con nombre del tool, parámetros y razón

Desafíos

Las reglas deben definirse explícitamente para cada operación protegida
No maneja lógica difusa o probabilística — las reglas son booleanas
Los casos límite requieren manejo explícito en las condiciones
Las reglas necesitan mantenimiento a medida que evoluciona la lógica de negocio

Mejores Prácticas

Define hooks para operaciones críticas: reservas, pagos, cancelaciones
Registra todas las violaciones de reglas con nombre del tool, parámetros y razón
Testea las reglas de forma independiente y exhaustiva
Combina con semantic tool selection (Parte 2) y multi-agent validation (Parte 5) para protección en capas

Siguiente Paso

Los guardrails simbólicos bloquean violaciones de reglas a nivel de tool. Pero bloquear detiene el flujo de trabajo — el usuario debe intervenir.

Parte 4: Runtime Guardrails muestra cómo Agent Control redirige a los agentes para que se autocorrijan en lugar de bloquearlos, completando flujos de trabajo sin intervención humana.

Conclusiones Clave

Los prompts son sugerencias: El LLM interpreta docstrings; puede alucinar cumplimiento con cualquier instrucción
Los hooks son aplicación forzada: BeforeToolCallEvent intercepta antes de la ejecución a nivel de framework — el LLM no puede anular un tool cancelado
Todas las operaciones inválidas bloqueadas: Cero cambios en tools, cero cambios en prompts, un hook agregado
Separación limpia: Los tools manejan operaciones; los hooks manejan la aplicación de restricciones
Auditable por diseño: Las violaciones de reglas son condiciones Python explícitas — testeables, registrables, trazables
Un hook, todos los tools: Un solo NeurosymbolicHook valida cada llamada a tool de forma centralizada

Ejecútalo Tú Mismo

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/04-neurosymbolic-demo
uv venv && uv pip install -r requirements.txt
uv run test_neurosymbolic_hooks.py

Puedes cambiar a cualquier proveedor soportado por Strands — consulta Strands Model Providers para la configuración.

Referencias

Investigación

Strands Agents

Código

Repositorio de Código

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube
Linktr

Elizabeth Fuentes LFollow

I help developers build production-ready AI applications through hands-on tutorials and open-source projects.

5 Techniques to Stop AI Agent Hallucinations in Production

Elizabeth Fuentes L — Mon, 30 Mar 2026 23:58:02 +0000

AI agent hallucinations occur when an LLM-powered agent fabricates data, selects the wrong tool, or ignores business rules during autonomous task execution. This post deploys 5 production-ready techniques to stop them — using managed hosting, serverless tools, database-driven guardrails, semantic tool routing, and a knowledge graph. Everything deploys as infrastructure as code.

TL;DR — 5 techniques, one production stack:

Graph-RAG on Neo4j AuraDB eliminates fabricated aggregations with Cypher queries
Semantic tool routing via AgentCore Gateway replaces custom FAISS indexes
Multi-agent validation on Lambda + DynamoDB catches errors single agents miss
Database-driven steering rules update agent behavior without redeploying
Hard hooks + soft steers separate financial constraints from operational adjustments

Every demo in this series on stopping AI agent hallucinations ran on a laptop. Hardcoded data, in-memory rules, a single user. The techniques worked, but the infrastructure did not scale. Moving to production means replacing local files with managed databases, in-process tools with serverless functions, and hardcoded rules with data you can update without redeploying.

This post walks through a production hotel booking agent that applies every technique from the series. We use Strands Agents for its native support of tool calling, hooks for guardrail enforcement, and MCP (Model Context Protocol) integration for connecting to Amazon Bedrock AgentCore Gateway. Similar patterns can be applied in LangGraph, AutoGen, CrewAI, or other agent frameworks.

Complexity note: This guide assumes familiarity with AWS CDK and core AWS services (Lambda, DynamoDB, S3). New to CDK? Start with the CDK Workshop first.

Working code: github.com/elizabethfuentes12/why-agents-fail-sample-for-amazon-agentcore

Prerequisites

You need the following:

AWS account with CDK bootstrapped in your target region
AWS CLI installed and configured
AWS CDK v2 (Cloud Development Kit) — infrastructure as code framework for deploying all resources
Python 3.11+ and uv package manager
OpenAI API key — the agent uses GPT-4o-mini as the LLM (Large Language Model) provider, swappable for Amazon Bedrock or other providers
Neo4j AuraDB Free account — only if deploying GraphRAG (Stack 2)

Series Overview

This is the final post in the series on stopping AI agent hallucinations:

RAG vs Graph-RAG: When Agents Hallucinate Answers — Knowledge graphs prevent hallucinated aggregations
Reduce Agent Errors and Token Costs with Semantic Tool Selection — Vector filtering reduces wrong tool choices
How to Stop AI Agents from Hallucinating Silently with Multi-Agent Validation — Cross-validation catches errors single agents miss
AI Agent Guardrails: Rules That LLMs Cannot Bypass — Symbolic rules enforced at the framework level
Runtime Guardrails for AI Agents — Steer, Don't Block — Agent self-corrects instead of failing
From Demo to Production (this post) — Deploy all techniques on AWS

Each post can be read independently. The repository contains all demos with runnable code.

How Does Each Anti-Hallucination Technique Map to Production?

The table below shows what each demo built locally and what replaces it in production:

Technique	Demo implementation	Production replacement
Graph-RAG (demo 01)	Local Neo4j Desktop + manual APOC scripts	Neo4j AuraDB Free + automated Lambda builder using SimpleKGPipeline
Semantic tool selection (demo 02)	In-memory FAISS (Facebook AI Similarity Search) index rebuilt on every run	AgentCore Gateway with MCP semantic routing — no custom index needed
Multi-agent validation (demo 03)	Hardcoded validator agents with in-memory state	`validate_booking_rules` Lambda backed by DynamoDB — same safety, lower latency
Neurosymbolic guardrails (demo 04)	Python hooks with hardcoded thresholds	Steering rules in DynamoDB — change rules without redeploying
Agent Control steering (demo 05)	Local Agent Control server with config files	STEER messages stored in DynamoDB rules — agent self-corrects from database-driven guidance

The anti-hallucination techniques remain the same. The infrastructure becomes managed, scalable, and updatable without code changes.

What Does the Production Architecture Look Like?

The production architecture runs on Amazon Bedrock AgentCore. Runtime hosts the agent code, and Gateway routes tool calls to AWS Lambda functions via MCP. Two independent AWS CDK stacks deploy the full infrastructure:

AgentCore Runtime — Hosts and runs your agent code in a managed environment
AgentCore Gateway — MCP-based semantic tool routing that connects agents to serverless functions

Stack 1 — HotelBookingAgentStack (deploy first, works on its own):

Resource	Purpose
3 Amazon DynamoDB tables	Hotels inventory, bookings, steering rules
AWS Secrets Manager	OpenAI API key stored securely
AgentCore Runtime	Managed agent hosting
AgentCore Gateway	MCP semantic routing to Lambda tools
7 AWS Lambda functions	search, book, get_booking, process_payment, confirm, cancel, validate

Stack 2 — GraphRAGStack (deploy when ready, adds FAQ capabilities):

Resource	Purpose
Amazon S3 bucket	300 hotel FAQ documents auto-uploaded during deploy
Lambda build_graph	Builds knowledge graph from documents using SimpleKGPipeline
Lambda query_knowledge_graph	Executes Cypher queries against Neo4j AuraDB
Neo4j AuraDB Free	Managed graph database ($0/month free tier)

How Do Steering Rules Work as Data Instead of Code?

In demo 04, business rules were hardcoded in Python. Changing a threshold meant changing code and redeploying. In production, rules live in DynamoDB. Update a row and the agent's behavior changes immediately.

Each rule has two messages: a fail_message that describes the violation and a steer_message that tells the agent how to self-correct:

{
    "rule_id": "max-guests",
    "action": "book",
    "condition_field": "guests",
    "operator": "gt",
    "threshold": 10,
    "fail_message": "Guest count exceeds maximum of 10",
    "steer_message": "Booking for 15 guests is not available, but you CAN book for up to 10. Adjust to 10 guests, proceed, and tell the user.",
    "enabled": true
}

The validate_booking_rules tool reads rules from DynamoDB before every booking action. When a rule is violated, the agent receives the STEER guidance and self-corrects, completing the task instead of blocking:

from strands import tool

@tool
def validate_booking_rules(
    action: str,
    guests: int = 0,
    check_in: str = "",
    check_out: str = "",
    booking_id: str = "",
) -> str:
    """Validate business rules BEFORE executing a booking action.
    ALWAYS call this before book_hotel, confirm_booking, or cancel_booking.
    Rules are loaded from the SteeringRules database, changeable without redeploying.
    """
    rules = _get_rules_for_action(action)       # DynamoDB scan
    context = _build_context(action, params)     # Derive nights, days_until_checkin
    violated = _evaluate_rules(rules, context)   # Symbolic evaluation

    if not violated:
        return f"PASS: All {len(rules)} rules passed for '{action}'. Proceed."

    lines = []
    for v in violated:
        lines.append(f"- {v['fail_message']}\n  STEER: {v['steer_message']}")
    return f"FAIL: {len(violated)} rule(s) violated:\n" + "\n".join(lines)

Changing a rule takes one command. No redeploy, no PR, takes effect immediately:

aws dynamodb update-item \
    --table-name HotelBookingAgentStack-SteeringRules \
    --key '{"rule_id": {"S": "max-guests"}}' \
    --update-expression "SET threshold = :t" \
    --expression-attribute-values '{":t": {"N": "8"}}'

The 6 steering rules deployed by default:

Rule	Action	What it catches	How the agent self-corrects
max-guests	book	More than 10 guests	Adjusts to 10, informs user
valid-dates	book	Check-out before check-in	Swaps dates, informs user
advance-booking	book	Same-day booking	Moves check-in to tomorrow
payment-before-confirm	confirm	Unpaid booking	Processes payment first
cancellation-window	cancel	Less than 48h to check-in	Suggests modification instead
already-cancelled	cancel	Booking already cancelled	Offers to create new booking

When Should You Block vs Steer?

Not every rule should be steerable. Payment before confirmation is a financial integrity constraint: the LLM must never bypass it, regardless of how it interprets a STEER message. The production agent uses a two-layer approach:

Layer 1 — Hard hooks (framework-level, cannot be bypassed):

from strands.hooks.events import BeforeToolCallEvent
from strands.hooks.registry import HookProvider, HookRegistry

class BookingGuardrailsHook(HookProvider):
    """Hard guardrails for financial and contractual constraints.
    The LLM cannot bypass these. They execute before the tool runs.
    """

    def register_hooks(self, registry: HookRegistry) -> None:
        registry.add_callback(BeforeToolCallEvent, self._validate)

    def _validate(self, event: BeforeToolCallEvent) -> None:
        tool_name = event.tool_use["name"]
        params = event.tool_use.get("input", {})

        if "confirm" in tool_name:
            booking = self._bookings.get_item(
                Key={"booking_id": params.get("booking_id", "")}
            ).get("Item")
            if booking and booking["status"] != "PAID":
                event.cancel_tool = (
                    "BLOCKED: Payment must be processed before confirmation. "
                    "Ask the user if they want to proceed with payment."
                )

Layer 2 — Soft steering (DynamoDB rules, agent self-corrects):

The validate_booking_rules tool shown above. Rules can be updated, disabled, or added without touching agent code.

How to decide which layer to use:

Layer	Mechanism	When to use	Can the LLM bypass it?
Hard hook	`event.cancel_tool` blocks execution	Financial, legal, compliance	No — framework intercepts before the tool runs
Soft steer	STEER message guides correction	Capacity limits, date adjustments, preferences	No bypass — but the agent adapts and completes the task
Prompt rule	System prompt instruction	Workflow order, communication style	Yes — the LLM may ignore under ambiguous input

Use hard hooks for rules where failure means financial or legal risk. Use soft steering for everything else. It reduces user friction without sacrificing safety.

How Does the MCP Integration with AgentCore Gateway Work?

The agent connects to AgentCore Gateway via MCP at startup and discovers all available tools at runtime, with no hardcoded tool list needed. In demo 02, we built a custom FAISS index to pre-filter tools by semantic similarity. In production, the Gateway replaces that entirely, routing each tool call to the right Lambda function based on semantic matching:

from bedrock_agentcore import BedrockAgentCoreApp
from strands import Agent
from strands.tools.mcp.mcp_client import MCPClient
from mcp.client.streamable_http import streamablehttp_client

# Using OpenAI-compatible interface via Strands SDK (not direct OpenAI usage)
from strands.models.openai import OpenAIModel

app = BedrockAgentCoreApp()

@app.entrypoint
def invoke(payload, context=None):
    """Entry point for AgentCore Runtime invocations."""
    model = OpenAIModel(
        model_id="gpt-4o-mini",
        client_args={"api_key": _openai_api_key},
    )
    hooks = [BookingGuardrailsHook()]

    # Gateway handles semantic tool routing via MCP
    mcp_client = MCPClient(lambda: streamablehttp_client(GATEWAY_URL))

    with mcp_client:
        tools = mcp_client.list_tools_sync()
        agent = Agent(
            model=model,
            tools=tools,
            system_prompt=SYSTEM_PROMPT,
            hooks=hooks,
        )
        prompt = payload.get("prompt", "")
        return str(agent(prompt))

The agent does not define tools inline. It connects to the Gateway, which discovers available Lambda functions and routes calls based on semantic matching. Adding a new tool (such as query_knowledge_graph) means registering it in the Gateway. No agent code changes.

How Does GraphRAG Work in Production?

Demo 01 demonstrates that traditional RAG (Retrieval-Augmented Generation) hallucinates when answering aggregation queries. "How many hotels have a pool?" gets a guess instead of a count. Graph-RAG eliminates this by executing Cypher queries on a knowledge graph that computes exact results.

In production, the knowledge graph runs on Neo4j AuraDB Free, a managed graph database with a $0/month free tier (200K nodes, no credit card required). The build pipeline is fully automated:

300 hotel FAQ documents upload to S3 during cdk deploy
build_graph Lambda reads documents, calls OpenAI to extract entities and relationships via SimpleKGPipeline, and loads them into Neo4j AuraDB
query_knowledge_graph Lambda executes Cypher queries and returns structured results
The query Lambda registers in AgentCore Gateway — the agent discovers it via MCP

Questions like "What amenities does the Grand Hotel have?" now traverse the graph (following connections between nodes) instead of guessing from text chunks.

The CDK stack supports two build modes via -c graph_mode=lite|full:

Build mode (`graph_mode`)	What it does	Documents	Build time
lite (default)	Uploads a subset of docs, builds graph in a single Lambda	30	~15 min
full	Uploads all docs, uses Step Functions to batch-process	300	~1-2 hours

cdk deploy GraphRAGStack                      # lite mode (30 docs)
cdk deploy GraphRAGStack -c graph_mode=full   # full mode (300 docs)

The booking agent works without GraphRAG. Deploy Stack 2 when hotel FAQ questions are frequent.

How to Deploy

Step 1 — Clone and install

git clone https://github.com/elizabethfuentes12/why-agents-fail-sample-for-amazon-agentcore
cd why-agents-fail-sample-for-amazon-agentcore/06-agentcore-production-demo

uv venv && uv pip install -r requirements.txt

Step 2 — Build and deploy

./create_deployment_package.sh

cd cdk
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cdk bootstrap   # Only needed once per AWS account/region
cdk deploy HotelBookingAgentStack

Step 3 — Store your OpenAI API key

Open AWS Secrets Manager Console
Find /HotelBookingAgentStack/openai-api-key
Set the secret value to your OpenAI API key

Step 4 — Seed hotel data

cd ..
AWS_DEFAULT_REGION=us-east-1 uv run python seed_data.py

Step 5 — Test

Open test_agent_local.ipynb and run all cells. The notebook tests every anti-hallucination layer — see the Try It Yourself section for the full scenario list.

Adding GraphRAG (optional, separate stack)

# 1. Create a free Neo4j AuraDB instance at neo4j.com/cloud/aura-free
# 2. Deploy the GraphRAG stack
cd cdk && INCLUDE_GRAPHRAG=1 cdk deploy GraphRAGStack

# 3. Store Neo4j credentials in Secrets Manager (4 secrets)
# 4. Build the knowledge graph
aws lambda invoke --function-name graphrag-build-graph \
    --region us-east-1 --cli-read-timeout 900 \
    /tmp/build-graph-output.json

# 5. Connect GraphRAG to the booking agent
cdk deploy HotelBookingAgentStack \
    -c graphrag_query_lambda_arn=<QueryLambdaArn from GraphRAGStack output>

Try It Yourself

The repository includes test_agent_local.ipynb — a notebook that tests every anti-hallucination layer against the deployed agent:

#	Scenario	Technique (from series)	What the agent does
1	Full booking flow	All layers	validate → book → pay → validate → confirm
2	15 guests (max 10)	Soft steering (demo 05)	Self-corrects to 10, informs user
3	Confirm without payment	Hard hook (demo 04)	Blocks confirmation, asks to pay first
4	Sold-out hotel	Grounded retrieval (demo 01)	Returns "no rooms available" from DynamoDB
5	City with no hotels	Grounded retrieval	Returns "no hotels found", no fabrication
6	Non-existent hotel	Grounded retrieval	Returns error, does not invent a hotel
7	Hotel amenities query	Graph-RAG (demo 01)	Cypher query returns real data from Neo4j

cd 06-agentcore-production-demo
uv venv && uv pip install -r requirements.txt
jupyter notebook test_agent_local.ipynb

Run all cells top to bottom. The notebook reads the AgentRuntimeArn from CloudFormation outputs automatically.

Built-in Observability

AgentCore provides built-in observability when you include the OpenTelemetry dependencies in your agent package. Add strands-agents[otel] and aws-opentelemetry-distro to your agent's requirements.txt:

strands-agents[openai,otel]>=1.27.0
aws-opentelemetry-distro>=0.7.0

With these dependencies, AgentCore automatically instruments Strands Agents — capturing invocation logs, tool call traces (which Lambda was called, input/output, latency), and error tracking (failed tool calls, guardrail blocks). You can monitor how the agent handles each anti-hallucination layer (steering, hooks, grounded retrieval) in Amazon CloudWatch without adding custom logging.

See the observability getting started guide for details.

How Fast Are the Anti-Hallucination Tools?

Tool latency matters in production agents — every tool call adds to the end-to-end response time. If a
validation check takes 2 seconds, users notice. We benchmarked the deployed Lambda functions to measure real
execution duration (from CloudWatch REPORT lines, not CLI round-trip time).

DynamoDB-Backed Tools

Function	Cold Start	Warm p50	Warm p90	Memory Used
`validate_booking_rules`	~550–620 ms	~25 ms	~105 ms	88 MB / 256 MB
`search_available_hotels`	~540–660 ms	~16 ms	~43 ms	88 MB / 256 MB

The steering rules check (validate_booking_rules) runs before every booking action. At ~25 ms warm, it adds negligible latency while preventing hallucinated bookings that violate business rules — a worthwhile trade-off vs. fixing bad bookings after the fact.

Neo4j AuraDB (GraphRAG)

Function	Cold Start	Warm p50	Warm p90	Memory Used
`query_knowledge_graph`	~1.9–2.0 s	~13 ms	~75 ms	122 MB / 256 MB

Neo4j AuraDB Free runs outside your VPC, but the neo4j Python driver reuses TCP connections across warm invocations. After the first call resolves credentials from Secrets Manager and establishes the connection, subsequent Cypher queries execute in 11–30 ms — comparable to DynamoDB.

DynamoDB vs Neo4j AuraDB: Side by Side

Metric	DynamoDB (steering rules)	Neo4j AuraDB (knowledge graph)
Warm p50	~16–25 ms	~13 ms
Warm p90	~43–105 ms	~75 ms
Cold start	~550–660 ms	~1.9–2.0 s
Driver reuse	boto3 (local)	neo4j driver (TCP keepalive)

Both backends deliver sub-100 ms warm latency, which means tool calls add minimal overhead to the agent's end-to-end response time. The cold start difference (~600 ms vs ~2 s) is only relevant for the first invocation after an idle period.

Benchmark methodology: 10 consecutive invocations per function using aws lambda invoke against the deployed stack in us-east-1. Duration values extracted from CloudWatch Lambda REPORT log lines. Cold start = Init Duration + Duration. Warm = Duration only (no Init).

5 Lessons from Moving Anti-Hallucination Techniques to Production

1. Separate hard blocks from soft steers. Payment-before-confirmation is a hard block: the agent cannot work around it. Guest count limits are a soft steer where the agent adjusts and completes the task. Mixing them in the same mechanism (all hooks or all prompts) either blocks too much or steers too little.

2. Store rules in a database, not in code. Business rules change more frequently than agent code. When the maximum guest count changes from 10 to 8, an operations team should update a DynamoDB item, not open a pull request and wait for CI/CD.

3. Deploy stacks independently. The booking agent works without GraphRAG. Independent stacks mean you can ship the core agent quickly and add capabilities when the use case demands it.

4. Test locally against the same data. The test_agent_local.ipynb notebook calls the same DynamoDB tables as production. Run all 7 test scenarios locally before deploying to AgentCore. The behavior is identical.

5. Keep tools as pure CRUD (Create, Read, Update, Delete). Lambda functions do one thing: read or write data. All business rule enforcement happens in the guardrail layers (hooks and validate tool). This keeps tools reusable and guardrails centralized.

Key Takeaways

Hard hooks block what the LLM must never bypass (financial, legal, compliance constraints)
Soft steering corrects what the agent can adapt (capacity, dates, preferences) — reducing user friction
DynamoDB steering rules let operations teams change agent behavior without code deploys
AgentCore Gateway replaces custom tool-selection indexes with managed MCP-based semantic routing
Graph-RAG on Neo4j AuraDB eliminates fabricated aggregations by computing exact results with Cypher queries

Clean Up

To avoid ongoing charges:

cd cdk
cdk destroy HotelBookingAgentStack
INCLUDE_GRAPHRAG=1 cdk destroy GraphRAGStack

Frequently Asked Questions

What is Amazon Bedrock AgentCore?

Amazon Bedrock AgentCore is a managed service for deploying and running AI agents. It provides two components: Runtime (hosts agent code) and Gateway (MCP-based tool routing that connects agents to serverless functions). AgentCore handles scaling, networking, and credential management so you do not need to build that infrastructure yourself.

Can I use Amazon Bedrock models instead of OpenAI?

Yes. The agent supports any provider compatible with Strands Agents, including Amazon Bedrock, Anthropic, or Ollama. Change the model in booking_agent.py. The anti-hallucination techniques work independently of the LLM provider.

What is the difference between hard hooks and soft steering?

Hard hooks use event.cancel_tool to block tool execution at the framework level. The LLM cannot bypass them. Soft steering returns a STEER message that instructs the agent how to self-correct: the agent adjusts parameters and completes the task. Use hard hooks for financial and compliance rules. Use soft steering for operational rules where the agent can adapt.

How much does this architecture cost?

All AWS services used (DynamoDB on-demand, Lambda, Secrets Manager) are eligible for the AWS Free Tier. Neo4j AuraDB Free has a $0/month tier. For current pricing details, see the AWS Pricing page. For a demo or low-traffic deployment, total infrastructure cost is minimal, excluding LLM API costs.

Do I need GraphRAG for the agent to work?

No. Stack 1 (HotelBookingAgentStack) deploys a fully functional booking agent with search, booking, payment, and validation tools. GraphRAG (Stack 2) adds hotel FAQ capabilities. Deploy it when you need answers about amenities, policies, and services.

Can I apply these patterns with other agent frameworks?

Yes. The guardrail patterns (database-driven rules, hard hooks, and soft steering) are framework-agnostic. Strands provides native hook and MCP support, but LangGraph, AutoGen, and CrewAI offer similar extension points for tool interception and pre-validation.

How does the agent discover tools via MCP at runtime?

The agent connects to AgentCore Gateway using MCPClient from Strands. On startup, it calls list_tools_sync() to discover all available Lambda tools registered in the Gateway. When the agent needs a tool, the Gateway routes the call based on semantic matching. There is no hardcoded tool list in the agent code. Adding a new tool means registering it in the Gateway; the agent discovers it automatically on the next invocation.

Research Background

The techniques in this series are grounded in recent research on AI agent reliability:

RAG-KG-IL: Multi-Agent Hybrid Framework for Reducing Hallucinations — Knowledge graphs reduce hallucinations compared to standalone LLMs
Internal Representations as Indicators of Hallucinations in Agent Tool Selection — Tool selection errors increase with tool count; semantic routing mitigates this
Teaming LLMs to Detect and Mitigate Hallucinations — Multi-agent validation detects errors that single agents miss
MetaRAG: Metamorphic Testing for Hallucination Detection — Hallucinations are inherent to LLMs without structured grounding

The complete code — CDK stacks, Lambda tools, agent runtime, steering rules, test notebook, and 300 hotel FAQ documents — is in the repository.

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube

[Boost]

Elizabeth Fuentes L — Tue, 24 Mar 2026 20:09:02 +0000

Hazel Saenz for AWS

Mar 24

Escala Inteligente con Microservicios Serverless

#serverless #microservices #architecture #aws

Comments

8 min read

Why AI Agents Fail: 3 Failure Modes That Cost You Tokens and Time

Elizabeth Fuentes L — Tue, 24 Mar 2026 02:14:33 +0000

AI agents don't fail like traditional software — they don't crash with a stack trace. They fail silently: returning incomplete answers, freezing on slow APIs, or burning tokens calling the same tool over and over. The agent appears to work, but the output is wrong, late, or expensive.

This series covers the three most common failure modes with research-backed solutions. Each technique has a runnable demo that measures the before/after difference.

Working code: github.com/aws-samples/sample-why-agents-fail

The demos use Strands Agents with OpenAI (GPT-4o-mini). The patterns are framework-agnostic — they apply to LangGraph, AutoGen, CrewAI, or any framework that supports tool calling and lifecycle hooks.

This Series: 3 Essential Fixes

Context Window Overflow — Memory Pointer Pattern for large data
MCP Tools That Never Respond — Async handleId pattern for slow external APIs
AI Agent Reasoning Loops — DebounceHook + clear tool states to block repeated calls

What Happens When Tool Outputs Overflow the Context Window?

Context window overflow occurs when a tool returns more data than the LLM can process — server logs, database results, or file contents that exceed the token limit. The agent does not crash with an error. It silently degrades: truncating data, losing context, or producing incomplete answers.

Research from IBM quantifies this: a Materials Science workflow consumed 20M tokens and failed. The same workflow with memory pointers used 1,234 tokens and succeeded.

The fix — Memory Pointer Pattern: Store large data in agent.state, return a short pointer to the context. The next tool resolves the pointer to access the full data:

from strands import tool, ToolContext

@tool(context=True)
def fetch_application_logs(app_name: str, tool_context: ToolContext, hours: int = 24) -> str:
    """Fetch logs. Stores large data as pointer to avoid context overflow."""
    logs = generate_logs(app_name, hours)  # Could be 200KB+

    if len(str(logs)) > 20_000:
        pointer = f"logs-{app_name}"
        tool_context.agent.state.set(pointer, logs)
        return f"Data stored as pointer '{pointer}'. Use analyze tools to query it."
    return str(logs)

@tool(context=True)
def analyze_error_patterns(data_pointer: str, tool_context: ToolContext) -> str:
    """Analyze errors — resolves pointer from agent.state."""
    data = tool_context.agent.state.get(data_pointer)
    errors = [e for e in data if e["level"] == "ERROR"]
    return f"Found {len(errors)} errors across {len(set(e['service'] for e in errors))} services"

The LLM never sees the 200KB — it only sees "Data stored as pointer 'logs-payment-service'" (52 bytes).

Why Strands Agents? The ToolContext API provides agent.state as a native key-value store scoped to each agent — no global dictionaries, no external infrastructure. For multi-agent workflows, invocation_state shares data across agents in a Swarm with the same API.

Metric	Without pointers	With Memory Pointers
Data in context	214KB (full logs)	52 bytes (pointer)
Agent behavior	Truncates or fails	Processes all data
Errors detected	Partial	Complete

Full demo: 01-context-overflow-demo — single-agent and multi-agent (Swarm) implementations with notebooks.

Why Do AI Agents Freeze When Calling External APIs?

AI agents freeze when MCP tools call slow or unresponsive external APIs. The agent blocks on the tool call, the user sees no progress, and after 7 seconds many implementations return a 424 error. MCP (Model Context Protocol) enables agents to call external tools, but does not handle timeout or retry by default.

The fix — Async handleId pattern: The tool returns immediately with a job ID. The agent polls a separate check_status tool:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("timeout-demo")
JOBS = {}

@mcp.tool()
async def start_long_job(task: str) -> str:
    """Return handle immediately — prevents timeout."""
    job_id = str(uuid.uuid4())[:8]
    JOBS[job_id] = {"status": "processing", "task": task}
    asyncio.create_task(_process_job(job_id))  # Background work
    return f"Job started. Handle: {job_id}. Use check_job_status to poll."

@mcp.tool()
async def check_job_status(job_id: str) -> str:
    """Poll job status — returns 'processing' or 'completed' with result."""
    job = JOBS.get(job_id)
    if not job:
        return f"FAILED: Job '{job_id}' not found"
    return f"{job['status'].upper()}: {job.get('result', 'Still processing...')}"

Scenario	Response time	UX
Fast API (1s)	3s total	OK
Slow API (15s)	18s blocked	Agent frozen
Failing API	424 error after 7s	Agent crashes
Async handleId	~4s (immediate + poll)	Agent responsive

Why Strands Agents? The MCPClient connects to any MCP server with one line. The agent discovers tools at runtime via list_tools_sync() — no hardcoded tool list. When the MCP server implements the async pattern, the agent automatically polls without extra orchestration code.

Full demo: 02-mcp-timeout-demo — local MCP server with all 4 scenarios and notebook.

Why Do AI Agents Repeat the Same Tool Call?

AI agent reasoning loops happen when the agent calls the same tool repeatedly with identical parameters, making no progress. The root cause is ambiguous tool feedback — responses like "more results may be available" make the agent think another call will produce better results. Research shows agents can loop hundreds of times without delivering an answer.

Fix 1 — Clear terminal states: Tools return explicit SUCCESS or FAILED instead of ambiguous messages:

# Ambiguous (causes loops)
return f"Found flights: {results}. More results may be available."

# Clear (agent stops)
return f"SUCCESS: Booked flight {conf_id} for {passenger}. Confirmation sent."

Fix 2 — DebounceHook: Detect and block duplicate tool calls at the framework level:

from strands.hooks.registry import HookProvider, HookRegistry
from strands.hooks.events import BeforeToolCallEvent

class DebounceHook(HookProvider):
    """Block duplicate tool calls in a sliding window."""
    def __init__(self, window_size=3):
        self.call_history = []
        self.window_size = window_size

    def register_hooks(self, registry: HookRegistry) -> None:
        registry.add_callback(BeforeToolCallEvent, self.check_duplicate)

    def check_duplicate(self, event: BeforeToolCallEvent) -> None:
        key = (event.tool_use["name"], json.dumps(event.tool_use.get("input", {})))
        if self.call_history.count(key) >= 2:
            event.cancel_tool = f"BLOCKED: Duplicate call to {event.tool_use['name']}"
        self.call_history.append(key)
        self.call_history = self.call_history[-self.window_size:]

Strategy	Tool calls	Result
Ambiguous feedback (baseline)	14 calls	No definitive answer
DebounceHook	12 calls (2 blocked)	Completes with blocks
Clear SUCCESS states	2 calls	Immediate completion

Why Strands Agents? The HookProvider API intercepts tool calls via BeforeToolCallEvent before they execute. Setting event.cancel_tool blocks execution at the framework level — the LLM cannot bypass it. This makes hooks composable: stack DebounceHook, LimitToolCounts, and custom validators on the same agent.

Full demo: 03-reasoning-loops-demo — all 4 scenarios with hooks and notebook.

Prerequisites

You need Python 3.9+, uv (a fast Python package manager), and an OpenAI API key.

git clone https://github.com/aws-samples/sample-why-agents-fail
cd sample-why-agents-fail/stop-ai-agents-wasting-tokens

# Pick any demo
cd 01-context-overflow-demo   # or 02-mcp-timeout-demo, 03-reasoning-loops-demo
uv venv && uv pip install -r requirements.txt
export OPENAI_API_KEY="your-key-here"

uv run python test_*.py

Each demo is self-contained with its own dependencies, test script, and Jupyter notebook.

Frequently Asked Questions

What are the most common AI agent failure modes?

The three most common failure modes are context window overflow (tool returns more data than the LLM can process), MCP tool timeouts (external APIs block the agent indefinitely), and reasoning loops (agent repeats the same tool call without progress). Each failure mode causes token waste and degrades response quality.

How do I reduce AI agent token costs?

The two most effective techniques are memory pointers and clear tool states. The Memory Pointer Pattern stores large tool outputs in external state and passes short references to the LLM context — reducing token usage from 200KB+ to under 100 bytes per tool call. Clear terminal states (SUCCESS/FAILED) in tool responses prevent the agent from retrying completed operations, which can reduce tool calls from 14 to 2.

Can I use these patterns with frameworks other than Strands Agents?

Yes. The Memory Pointer Pattern works with any framework that supports tool context (passing state between tools). The async handleId pattern is an MCP server design pattern — it works with any MCP-compatible agent. DebounceHook requires lifecycle hooks, which are available in LangGraph, AutoGen, and CrewAI with different APIs.

References

Research

Solving Context Window Overflow in AI Agents — IBM Research, Nov 2025
Towards Effective GenAI Multi-Agent Collaboration — Amazon, Dec 2024
Resilient AI Agents With MCP — Octopus, May 2025
Language models can overthink — The Decoder, Jan 2025

Implementation

Strands Agent State — ToolContext and agent.state
Strands MCP Tools — Connect any MCP server
Strands Hooks — Lifecycle events and tool cancellation

Which failure mode have you hit in your agents? Share in the comments.

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube
Linktr

Elizabeth Fuentes LFollow

I help developers build production-ready AI applications through hands-on tutorials and open-source projects.

Reduce errores y costos de tokens en agentes con seleccion semantica de herramientas

Elizabeth Fuentes L — Mon, 23 Mar 2026 16:25:05 +0000

Cuando los agentes de IA tienen muchas herramientas similares, a menudo seleccionan la incorrecta y consumen tokens excesivos al procesar todas las descripciones de herramientas.

Este articulo demuestra como la seleccion semantica de herramientas filtra tools antes del procesamiento del agente, mejorando la precision y reduciendo costos de tokens. La demostracion utiliza Strands Agents y FAISS para filtrar 29 herramientas y quedarse con las 3 mas relevantes.

Esta demo utiliza Strands Agents. Patrones similares se pueden aplicar en LangGraph, AutoGen u otros frameworks de agentes.

Vision general de la serie

Esta es la Parte 2 de una serie sobre como detener las hallucinations en agentes de IA:

RAG vs Graph-RAG — Knowledge graphs para prevenir hallucinations
Seleccion semantica de herramientas (este articulo) — Filtrado de tools basado en vectores
Validacion multi-agente — Deteccion de hallucinations basada en equipos
AI Agent Guardrails — Aplicacion de razonamiento simbolico
Runtime Guardrails — Controles autocorrectivos

Configuracion

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/02-semantic-tools-demo
uv venv && uv pip install -r requirements.txt

El problema dual: errores + desperdicio de tokens

Escenario del problema

Un agente de viajes con 29 herramientas similares (search_hotels, search_flights, search_hotel_reviews, etc.) recibe la consulta: "How much does Hotel Marriott cost?"

El agente puede seleccionar get_hotel_details() en lugar de get_hotel_pricing() — una seleccion incorrecta de tool. Este error consume 4,500 tokens procesando las descripciones de las 29 herramientas.

Causas raiz

Nombres de herramientas similares causan confusion
Las herramientas genericas se usan en exceso
Mas herramientas aumentan la probabilidad de hallucination

Contexto de investigacion

"Las hallucinations en la seleccion de herramientas aumentan con la cantidad de tools. Sistemas en produccion reportan una reduccion del 89% en tokens con seleccion semantica de herramientas."

La solucion: seleccion semantica de herramientas con FAISS

Modos de falla de agentes a escala

La investigacion identifica cinco modos criticos de falla:

Errores de seleccion de funcion — Llamar a tools inexistentes
Errores de parametros — Argumentos mal formados
Errores de completitud — Parametros requeridos faltantes
Comportamiento de bypass de tools — Generar respuestas en lugar de llamar herramientas
Desbordamiento de contexto — Desperdicio de tokens al procesar todas las descripciones

Analisis costo-beneficio

Cada llamada al LLM envia las descripciones de las 29 herramientas
En un flujo de 50 pasos: 29 tools x 50 llamadas
Genera un desperdicio significativo de tokens y retrasos en el procesamiento

La investigacion muestra hasta un 86.4% de precision previniendo hallucinations en la seleccion de herramientas en sistemas de produccion.

Demostracion: tres enfoques

Prueba 1: Enfoque tradicional (las 29 herramientas)

from strands import Agent
# Using OpenAI-compatible interface via Strands SDK (not direct OpenAI usage)
from strands.models.openai import OpenAIModel
from enhanced_tools import ALL_TOOLS

for query, expected in TESTS:
    agent = Agent(tools=ALL_TOOLS, system_prompt=PROMPT, model=MODEL)
    tools, tokens = run_and_capture_with_tokens(agent, query)

Resultado: ~1,557 tokens promedio por consulta con precision variable.

Prueba 2: Enfoque semantico (top-3 herramientas filtradas)

El agente recibe unicamente las 3 herramientas mas relevantes por consulta.

Construccion del indice FAISS

import faiss
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def build_index(tools):
    """Build FAISS index from tool docstrings"""
    texts = [f"{t.__name__}: {t.__doc__}" for t in tools]
    embeddings = model.encode(texts)

    index = faiss.IndexFlatL2(embeddings.shape[1])
    index.add(embeddings.astype('float32'))
    return index

def search_tools(query: str, top_k: int = 3):
    """Find most relevant tools using FAISS"""
    emb = model.encode([query])
    _, indices = index.search(emb.astype('float32'), top_k)
    return [tools[i] for i in indices[0]]

El modelo all-MiniLM-L6-v2 es ligero (22M parametros, 384 dimensiones), optimizado para similitud semantica y funciona de manera eficiente en CPUs.

Filtrado en tiempo de ejecucion

from strands import Agent
# Using OpenAI-compatible interface via Strands SDK (not direct OpenAI usage)
from strands.models.openai import OpenAIModel

for query, expected in TESTS:
    selected = search_tools(query, top_k=3)
    selected_names = [t.__name__ for t in selected]

    agent = Agent(tools=selected, system_prompt=PROMPT, model=MODEL)
    tools, tokens = run_and_capture_with_tokens(agent, query)

Resultado: ~275 tokens promedio por consulta — una reduccion desde 1,557 tokens.

Dato clave: el agente nunca procesa las otras 26 herramientas. Estas tools permanecen en el sistema pero nunca entran en la ventana de contexto del agente.

Prueba 3: Semantico + memoria (agente unico)

Para conversaciones multi-turno en produccion, manteniendo la memoria mientras se intercambian herramientas dinamicamente.

Intercambio dinamico de herramientas

def swap_tools(agent, new_tools):
    """Swap tools in a live agent without losing conversation memory."""
    reg = agent.tool_registry
    reg.registry.clear()
    reg.dynamic_tools.clear()
    for t in new_tools:
        reg.register_tool(t)

Implementacion completa

from strands import Agent
# Using OpenAI-compatible interface via Strands SDK (not direct OpenAI usage)
from strands.models.openai import OpenAIModel
from enhanced_tools import ALL_TOOLS
from registry import build_index, search_tools, swap_tools

initial_tools = search_tools(TESTS[0][0], top_k=3)
memory_agent = Agent(tools=initial_tools, system_prompt=PROMPT, model=MODEL)

for query, expected in TESTS:
    selected = search_tools(query, top_k=3)
    swap_tools(memory_agent, selected)
    tools, tokens = run_and_capture_with_tokens(memory_agent, query)

Resultado: Mas tokens que el enfoque solo semantico (debido al contexto acumulado), pero significativamente menos comparado con el metodo tradicional, manteniendo el historial completo de la conversacion.

Por que funciona

Strands llama a tool_registry.get_all_tools_config() en cada ciclo del event loop, detectando automaticamente los cambios en tiempo de ejecucion.

Ventajas:

Cero perdida de conversacion: agent.messages se preserva entre intercambios de tools
No requiere recrear el agente
Flexibilidad en tiempo de ejecucion para necesidades dinamicas de herramientas
Listo para produccion en conversaciones largas

Contexto de investigacion y rendimiento en el mundo real

La demo controlada logro precision perfecta en 13 consultas. Sin embargo, los sistemas de produccion reales con cientos de herramientas y consultas ambiguas muestran resultados diferentes.

"La investigacion muestra que en sistemas de produccion con cientos de herramientas, la seleccion semantica de tools alcanza hasta un 86.4% de precision en la deteccion y prevencion de hallucinations en la seleccion de herramientas."

Esto reduce significativamente los errores comparado con los enfoques tradicionales (que caen por debajo del 50% de precision con mas de 100 tools), pero sigue siendo un desafio en dominios con semantica de herramientas superpuesta.

Siguiente paso

La seleccion semantica de herramientas reduce los errores de seleccion de tools y los costos de tokens. Sin embargo, los agentes aun pueden generar hallucinations sobre el exito de operaciones (por ejemplo, confirmar reservaciones sin procesar pagos o ignorar reglas de negocio).

Parte 3: Validacion multi-agente demuestra como equipos especializados de agentes (Executor -> Validator -> Critic) detectan hallucinations antes de que lleguen a los usuarios.

Esta capacidad esta disponible de forma nativa a traves de Amazon Bedrock Agentcore Gateway.

Puntos clave

Problema dual: Errores en la seleccion de herramientas Y desperdicio de tokens
Reduccion significativa de errores: Menos herramientas = menos selecciones incorrectas
89% de ahorro en tokens: Reduciendo de 29 a 3 herramientas por llamada
Implementacion sencilla: ~20 lineas de codigo con FAISS
Integracion con Strands: El decorador @tool y la carga dinamica permiten el filtrado semantico con intercambio de herramientas en tiempo de ejecucion

Ejecutalo tu mismo

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/02-semantic-tools-demo
uv venv && uv pip install -r requirements.txt
uv run test_semantic_tool_selection.py

Puedes cambiar a cualquier proveedor soportado por Strands — consulta Strands Model Providers para la configuracion.

Referencias

Investigacion

Internal Representations as Indicators of Hallucinations
Solving Context Window Overflow — Reduccion de tokens 7x
Semantic Tool Selection in Practice — Reduccion del 89%

Strands Agents

Codigo

Repositorio de codigo

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube

[Boost]

Elizabeth Fuentes L — Sat, 21 Mar 2026 15:03:44 +0000

Elizabeth Fuentes L for AWS Español

Mar 18

Detén las Hallucinations en Agentes de IA: 4 Técnicas Esenciales

#agents #ai #python #tutorial

Comments

6 min read

[Boost]

Elizabeth Fuentes L — Wed, 18 Mar 2026 01:08:21 +0000

Cold Starts en AWS Lambda: Qué son, por qué ocurren y cómo solucionarlos

Hazel Saenz for AWS ・ Mar 16

#lambda #performance #español #aws

RAG vs GraphRAG: Cuando los Agentes Alucinan Respuestas

Elizabeth Fuentes L — Wed, 18 Mar 2026 01:05:06 +0000

Un agent que alucina durante la ejecucion es catastrofico: podria fabricar parametros de API, inventar confirmaciones de exito tras fallos, o ejecutar acciones basadas en creencias falsas.

Este articulo explora como el Retrieval-Augmented Generation (RAG) tradicional provoca que los AI agents alucinen estadisticas y agregaciones, en contraste con GraphRAG usando knowledge graphs de Neo4j. Demostramos las diferencias a traves de un agent de reservas de viajes que compara RAG (FAISS) frente a GraphRAG sobre 300 documentos FAQ de hoteles.

Esta demo utiliza Strands Agents. Patrones similares se pueden aplicar en LangGraph, AutoGen u otros frameworks de agents.

Vision General de la Serie

Esta es la Parte 1 de una serie sobre como detener las hallucinations en AI agents:

RAG vs Graph-RAG (este articulo) — Los knowledge graphs previenen agregaciones alucinadas
Semantic Tool Selection — El filtrado vectorial reduce las elecciones incorrectas de tools
Multi-Agent Validation — Los swarms detectan hallucinations antes de que lleguen a los usuarios
AI Agent Guardrails — Reglas simbolicas que el LLM no puede eludir
Runtime Guardrails — Dirigir en lugar de bloquear

El Problema: Tres Tipos de Hallucinations en RAG

La investigacion (MetaRAG: Metamorphic Testing for Hallucination Detection) identifica tres tipos de hallucination en sistemas RAG:

Estadisticas fabricadas — Los sistemas RAG producen numeros estimados en lugar de valores calculados. El LLM adivina agregaciones a partir de unos pocos documentos recuperados en vez de calcularlas sobre el conjunto completo de datos.
Recuperacion incompleta — La busqueda vectorial recupera los top-k fragmentos mas similares, pero los datos necesarios para una respuesta pueden estar dispersos en muchos documentos. El LLM llena los vacios con fabricaciones que suenan plausibles.
Fabricacion fuera de dominio — Cuando no existen datos relevantes, RAG devuelve los resultados de apariencia mas similar de todos modos. El LLM trata estas coincidencias parciales como contexto valido y fabrica una respuesta con confianza.

La Solucion: GraphRAG con Knowledge Graphs

Los knowledge graphs resuelven estos problemas a nivel arquitectonico:

Las agregaciones son calculadas por el motor de base de datos (AVG, COUNT, SUM) — no adivinadas por el LLM
Las relaciones son aristas explicitas en el grafo — no inferidas por similitud de texto
Los datos faltantes devuelven resultados vacios — el LLM recibe "no se encontraron resultados" en lugar de coincidencias parciales de las cuales alucinar

La diferencia clave: RAG siempre devuelve algo. GraphRAG devuelve resultados vacios cuando los datos no existen.

Requisitos Previos

Python 3.9+
Neo4j ejecutandose localmente con el plugin APOC
Clave de API de OpenAI (o cualquier proveedor de modelos compatible con Strands)

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/01-faq-graphrag-demo
uv venv && uv pip install -r requirements.txt

Implementacion

Agent 1: RAG Tradicional (FAISS)

from strands import Agent, tool

@tool
def search_faqs(query: str) -> str:
    """Search hotel FAQs using vector similarity."""
    query_embedding = embed_model.encode([query])
    distances, indices = index.search(query_embedding.astype('float32'), 3)
    return "\n".join([documents[idx]['text'][:500] for idx in indices[0]])

rag_agent = Agent(tools=[search_faqs], model=model)

El agent RAG recupera los 3 documentos mas similares y permite que el LLM los resuma. No puede agregar sobre el conjunto completo de datos, recorrer relaciones ni reconocer cuando los datos estan ausentes.

Agent 2: GraphRAG (Neo4j + Cypher)

@tool
def query_knowledge_graph(cypher_query: str) -> str:
    """Execute a Cypher query against the hotel knowledge graph.

    Node labels: Hotel, Room, Amenity, Policy, Service
    Relationships: (Hotel)-[:HAS_ROOM]->(Room), (Hotel)-[:OFFERS_AMENITY]->(Amenity),
                   (Hotel)-[:HAS_POLICY]->(Policy), (Hotel)-[:PROVIDES_SERVICE]->(Service)

    Properties:
    - Hotel: name, location, rating, price_range
    - Room: type, capacity, price_per_night
    - Amenity: name, category
    """
    with driver.session() as session:
        result = session.run(cypher_query)
        records = list(result)
        if not records:
            return "No results found."
        return f"Found {len(records)} results:\n" + "\n".join(str(dict(r.items())) for r in records[:15])

graph_agent = Agent(tools=[query_knowledge_graph], model=model)

El schema en el docstring habilita Text2Cypher — el LLM genera consultas Cypher precisas en lugar de adivinar a partir de fragmentos de texto.

Resultados: 4 Escenarios de Prueba

Prueba 1: Agregacion — "What is the average hotel rating in Paris?"

Enfoque	Resultado	Por que
RAG	4.7 (de solo 2 documentos)	Recupero 2 documentos similares y promedio esas calificaciones
GraphRAG	4.7 (a nivel de base de datos) ✅	`MATCH (h:Hotel {location:'Paris'}) RETURN avg(h.rating)` — calculado sobre todos los hoteles

RAG tuvo suerte aqui — mismo numero, pero de 2 documentos en vez del conjunto completo de datos. Con datos diferentes, esto divergiria.

Prueba 2: Conteo — "How many hotels have a swimming pool?"

Enfoque	Resultado	Por que
RAG	"No tengo los datos" ❌	Los 3 documentos principales no mencionaban piscinas; no pudo contar entre 300 documentos
GraphRAG	"133 hoteles" ✅	`MATCH (h)-[:OFFERS_AMENITY]->(a {name:'pool'}) RETURN count(h)`

Prueba 3: Multi-hop — "What room types does the highest-rated hotel offer?"

Enfoque	Resultado	Por que
RAG	Incompleto ⚠️	Encontro informacion del hotel pero no pudo recorrer hasta los datos de habitaciones
GraphRAG	Completo con datos de habitaciones ✅	`MATCH (h:Hotel) WITH h ORDER BY h.rating DESC LIMIT 1 MATCH (h)-[:HAS_ROOM]->(r) RETURN r`

Prueba 4: Fuera de dominio — "Find hotels in Antarctica"

Enfoque	Resultado	Por que
RAG	Fabrico "Estaciones de Investigacion" como hoteles ❌	Devolvio documentos de apariencia similar sobre alojamiento en clima frio
GraphRAG	"No se encontraron hoteles" ✅	`MATCH (h:Hotel {location:'Antarctica'}) RETURN h` — resultado vacio

Cuando Usar Cada Enfoque

Criterio	RAG	GraphRAG
Estructura de datos	Texto no estructurado, documentos	Estructurado, con relaciones
Tipo de consulta	Similitud semantica, busqueda difusa	Agregaciones precisas, recorridos
Datos faltantes	Devuelve resultados similares (riesgo de hallucination)	Devuelve vacio (fallo honesto)
Complejidad de configuracion	Menor (embeddings + vector store)	Mayor (knowledge graph + schema)
Ideal para	Busqueda de contenido, Q&A sobre documentos	Analitica, razonamiento multi-hop, verificacion

Conclusiones Clave

RAG siempre devuelve algo — incluso cuando los datos no existen, creando oportunidades de hallucination
GraphRAG devuelve resultados vacios cuando los datos estan ausentes — forzando respuestas honestas de "No lo se"
Las agregaciones deben ser calculadas por la base de datos, no adivinadas por el LLM
El schema en el docstring del tool habilita Text2Cypher — consultas precisas en lugar de similitud de texto
Ambos enfoques tienen su lugar — usa RAG para busqueda semantica, GraphRAG para recuperacion precisa de datos

Siguiente Paso

GraphRAG previene estadisticas y agregaciones alucinadas. Pero los agents aun pueden seleccionar el tool incorrecto o desperdiciar tokens procesando descripciones de tools irrelevantes.

Parte 2: Semantic Tool Selection muestra como el filtrado vectorial reduce los errores de seleccion de tools y los costos de tokens.

Pruebalo Tu Mismo

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations/01-faq-graphrag-demo
uv venv && uv pip install -r requirements.txt
# Start Neo4j with APOC plugin first
uv run test_graphrag_hallucinations.py

Puedes cambiar a cualquier proveedor compatible con Strands — consulta Strands Model Providers para la configuracion.

Referencias

Investigacion

Strands Agents

Codigo

Repositorio de Codigo

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube

Detén las Hallucinations en Agentes de IA: 4 Técnicas Esenciales

Elizabeth Fuentes L — Wed, 18 Mar 2026 00:13:33 +0000

4 técnicas para detener las hallucinations en AI agents: Graph-RAG para recuperación precisa de datos, selección semántica de tools para elegir la herramienta correcta, guardrails neurosimbólicos para aplicar reglas de negocio, y validación multi-agent para detección de errores.

Objetivos de Aprendizaje

Cómo Graph-RAG previene hallucinations estadísticas con datos estructurados
Por qué la selección semántica de tools mejora la precisión y reduce costos de tokens
Cómo los guardrails neurosimbólicos con Strands Agents bloquean operaciones inválidas
Cómo la validación multi-agent detecta hallucinations antes de que lleguen a los usuarios

Esta demo utiliza Strands Agents. Patrones similares pueden aplicarse en LangGraph, AutoGen u otros frameworks de agents.

Repositorio y Prerequisitos

Clonar:

git clone https://github.com/aws-samples/sample-why-agents-fail
cd stop-ai-agent-hallucinations

Requisitos:

Python 3.9+
Acceso a un LLM (Amazon Bedrock, OpenAI, Anthropic u Ollama)
Credenciales de AWS configuradas (si se usa Bedrock)
Comprensión básica de AI agents y tool calling
Bibliotecas clave: Strands Agents, Neo4j, FAISS, SentenceTransformers

Técnica 1: Graph-RAG para Recuperación Precisa de Datos

Problema: El RAG tradicional recupera fragmentos de texto en lugar de ejecutar cálculos, lo que produce tres tipos de hallucination:

Estadísticas fabricadas a partir de agregaciones adivinadas
Recuperación incompleta cuando los datos están dispersos
Fabricación fuera de dominio cuando no existen datos relevantes

Solución: Los knowledge graphs proporcionan datos estructurados y verificables, donde las agregaciones son calculadas por la base de datos en lugar de ser adivinadas por los LLMs.

Código de Comparación

from strands import Agent, tool

# RAG Agent — vector similarity search
@tool
def search_faqs(query: str) -> str:
    """Search hotel FAQs using vector similarity."""
    query_embedding = embed_model.encode([query])
    distances, indices = index.search(query_embedding.astype('float32'), 3)
    return "\n".join([documents[idx]['text'][:500] for idx in indices[0]])

# Graph-RAG Agent — Cypher queries on knowledge graph
@tool
def query_knowledge_graph(cypher_query: str) -> str:
    """Execute a Cypher query against the hotel knowledge graph.
    Node labels: Hotel, Room, Amenity, Policy, Service
    Relationships: (Hotel)-[:HAS_ROOM]->(Room), (Hotel)-[:OFFERS_AMENITY]->(Amenity)
    """
    with driver.session() as session:
        result = session.run(cypher_query)
        records = list(result)
        if not records:
            return "No results found."
        return f"Found {len(records)} results:\n" + "\n".join(str(dict(r.items())) for r in records[:15])

rag_agent = Agent(tools=[search_faqs], model=model)
graph_agent = Agent(tools=[query_knowledge_graph], model=model)

Resultados de la Demo

Tipo de Consulta	RAG	Graph-RAG
Agregación: "¿Calificación promedio en París?"	⚠️ Calcula solo con 2 documentos	✅ AVG() nativo sobre todos
Conteo: "¿Hoteles con piscina?"	❌ "No tengo los datos"	✅ Preciso: 133
Multi-hop: "¿Tipos de habitación del mejor hotel?"	❌ No puede recorrer relaciones	✅ Recorrido Hotel → Room
Fuera de dominio: "Hoteles en la Antártida"	❌ Fabrica respuestas	✅ Honesto: "No hay hoteles"

Hallazgo Clave: Graph-RAG reduce las hallucinations porque devuelve resultados vacíos en lugar de respuestas fabricadas cuando los datos no están disponibles.

Técnica 2: Selección Semántica de Tools para Elegir la Herramienta Correcta

Problema: "Las hallucinations en tool-calling aumentan con la cantidad de tools." Cuando los agents tienen muchas tools similares, exhiben errores de selección de función, errores de idoneidad, errores de parámetros, errores de completitud y comportamiento de evasión de tools.

Problema Dual:

❌ Riesgo de hallucination: Más tools = más selecciones inapropiadas
❌ Desperdicio de tokens: 31 tools ≈ 4,500 tokens por consulta

Solución: Filtrado Semántico de Tools

Filtra las tools antes de que el agent las vea usando similitud vectorial.

from sentence_transformers import SentenceTransformer
import faiss

# Build index once
model = SentenceTransformer('all-MiniLM-L6-v2')
tool_embeddings = model.encode([tool.description for tool in ALL_TOOLS])
index = faiss.IndexFlatL2(384)
index.add(tool_embeddings)

# Filter per query
query_embedding = model.encode([query])
distances, indices = index.search(query_embedding, k=3)
relevant_tools = [ALL_TOOLS[i] for i in indices[0]]

Intercambio Dinámico de Tools con Strands Agents

from strands import Agent

# Create agent once
agent = Agent(tools=[...], model=model)

# Swap tools per query without losing conversation history
for query in conversation:
    relevant_tools = search_tools(query, top_k=3)
    agent.tool_registry.registry.clear()
    for tool in relevant_tools:
        agent.tool_registry.register_tool(tool)

    response = agent(query)  # agent.messages preserved

Resultados en 29 Consultas de Viaje

Las pruebas muestran mejoras significativas:

86.4% de reducción en errores de selección de tools
89% de reducción en costos de tokens
Precisión mantenida con menos tools visibles

Técnica 3: Guardrails Neurosimbólicos para AI Agents

Problema: Investigaciones demuestran que los agents generan hallucinations cuando las reglas de negocio se expresan únicamente en prompts de lenguaje natural. El prompt engineering no puede aplicar restricciones porque los prompts son sugerencias, no reglas ejecutables.

Solución: Usar hooks de Strands Agents para interceptar llamadas a tools antes de su ejecución y aplicar reglas simbólicas a nivel del framework.

Implementación

from strands import Agent, tool
from strands.hooks import HookProvider, HookRegistry, BeforeToolCallEvent

# Define symbolic rules
BOOKING_RULES = [
    Rule(
        name="max_guests",
        condition=lambda ctx: ctx.get("guests", 1) <= 10,
        message="Maximum 10 guests per booking"
    ),
]

# Create validation hook
class NeurosymbolicHook(HookProvider):
    def register_hooks(self, registry: HookRegistry) -> None:
        registry.add_callback(BeforeToolCallEvent, self.validate)

    def validate(self, event: BeforeToolCallEvent) -> None:
        ctx = {"guests": event.tool_use["input"].get("guests", 1)}
        passed, violations = validate(BOOKING_RULES, ctx)

        if not passed:
            event.cancel_tool = f"BLOCKED: {', '.join(violations)}"

# Clean tool (no validation logic)
@tool
def book_hotel(hotel: str, guests: int = 1) -> str:
    """Book a hotel room."""
    return f"SUCCESS: Booked {hotel} for {guests} guests"

# Attach hook to agent
hook = NeurosymbolicHook()
agent = Agent(tools=[book_hotel], hooks=[hook])

# Test
query = "Book hotel for 15 guests"
result = agent(query)  # ✅ Hook blocks before tool executes

Por Qué los Hooks de Strands Destacan

API sencilla: Implementar HookProvider y registrar callbacks
Validación centralizada: Un solo hook valida todas las tools
Tools limpias: Sin lógica de validación mezclada con lógica de negocio
Tipado seguro: Objetos de eventos fuertemente tipados
El LLM no puede evadir: Las reglas se aplican antes de la ejecución de la tool

Efectividad

Escenario	Prompt Engineering	Neurosimbólico con Hooks
Parámetros Inválidos	❌ Acepta	✅ Bloquea
Prerequisitos Faltantes	⚠️ A veces detecta	✅ Siempre bloquea
Evasión de Reglas	❌ Posible	✅ Imposible

Técnica 4: Validación Multi-Agent para Detección de Errores

Problema: Los agents individuales operan en aislamiento. Cuando generan hallucinations, no existe un mecanismo para detectar errores antes de que lleguen a los usuarios.

Solución: Múltiples agents especializados se validan mutuamente mediante debate estructurado, utilizando Strands Swarm para transferencias autónomas con contexto compartido.

Implementación

from strands import Agent
from strands.multiagent import Swarm

# Define specialized agents
executor = Agent(
    name="executor",
    tools=ALL_TOOLS,
    system_prompt="Execute requests, then hand off to validator"
)

validator = Agent(
    name="validator",
    system_prompt="Check for hallucinations. Say VALID or HALLUCINATION"
)

critic = Agent(
    name="critic",
    system_prompt="Final review. Say APPROVED or REJECTED"
)

# Create swarm - agents hand off autonomously
swarm = Swarm(
    [executor, validator, critic],
    entry_point=executor,
    max_handoffs=5
)

result = swarm("Book grand_hotel for John")

Comparación de Rendimiento

Enfoque	Detección de Hallucinations	Precisión	Latencia
Agent Individual	❌ Ninguna	⚠️ Fabrica	✅ Rápido
Multi-Agent	✅ Detecta errores	✅ Valida	⚠️ Más lento

Ejemplo: Cuando el executor intenta reservar "the_ritz_paris" (no existe), el validator detecta el hotel inválido y el critic devuelve FAILED en lugar de fabricar una alternativa.

Combinando Técnicas para Producción

Estas técnicas se complementan entre sí:

Graph-RAG asegura la precisión de los datos
Selección semántica de tools reduce errores de tools y costos de tokens
Reglas neurosimbólicas aplican restricciones de negocio
Validación multi-agent detecta las hallucinations restantes

Por Qué Strands Agents Python SDK

Gestión dinámica de tools sin perder el estado de la conversación
Multi-agent nativo con Swarm manejando transferencias
Validación a nivel de tool ejecutándose antes de que el LLM vea los resultados
Flexibilidad de modelos (Bedrock, OpenAI, Anthropic, Ollama)
Listo para producción en despliegues AWS

Conclusiones Clave

Las hallucinations son inevitables — enfócate en detección y mitigación, no en eliminación
Graph-RAG para precisión cuando necesitas cálculos exactos o relaciones
Filtrado semántico esencial con más de 10 tools similares
Las reglas simbólicas aplican cumplimiento de negocio donde el prompt engineering falla
La validación multi-agent detecta errores que los agents individuales no captan
Strands Agents permite despliegues en producción con tools dinámicas y soporte nativo multi-agent

Referencias

Gracias!

🇻🇪🇨🇱 Dev.to Linkedin GitHub Twitter Instagram Youtube
Linktr

Elizabeth Fuentes LFollow

I help developers build production-ready AI applications through hands-on tutorials and open-source projects.