Forem: refaat Al Ktifan

Modern TypeScript Backends: Hono, ElysiaJS, and What Comes After NestJS

refaat Al Ktifan — Fri, 01 May 2026 09:00:10 +0000

Why We Stopped Starting New Projects with NestJS

NestJS is a great framework. We've built enterprise systems on it (Vendure is NestJS under the hood, and we love Vendure). But for new standalone backend services, we've moved to Hono. Not because NestJS is bad. Because Hono is better for the way we work now.

The shift happened gradually. We started a new API service, considered NestJS, and asked ourselves: do we need decorators, modules, providers, and the Angular-inspired dependency injection container for a 15-endpoint API? The answer was no. We needed a fast HTTP framework that runs on any runtime, has good TypeScript support, and doesn't prescribe an architecture.

Hono delivered that. Then we tried ElysiaJS on Bun for a performance-critical service. That delivered something different. This article is an honest comparison from someone who runs all three in production.

For broader context on how we approach software engineering, that guide covers our principles. For how these frameworks fit into commerce architecture, see our Vendure production guide.

The Three Frameworks

Feature	NestJS	Hono	ElysiaJS
Runtime	Node.js	Node, Bun, Deno, Cloudflare Workers, Lambda, Vercel	Bun (primarily)
Architecture	Opinionated (modules, controllers, providers)	Minimal (routes, middleware)	Minimal (routes, plugins)
Type safety	Decorators + runtime validation	Middleware-based, manual	Built-in validation (typebox)
DI container	Built-in (Angular-style)	None (bring your own)	None (plugins)
Bundle size	Large (~50MB node_modules)	Tiny (~14KB core)	Small (~2MB with Bun)
Cold start	Slow (2-5s on Lambda)	Fast (< 100ms on Lambda)	Fast (< 50ms on Bun)
Middleware ecosystem	Large (passport, class-validator, etc.)	Growing (auth, cors, jwt, etc.)	Growing (swagger, jwt, etc.)
Learning curve	Steep (modules, decorators, DI, guards, pipes)	Minimal (express-like)	Minimal (express-like with validation)
Best for	Large enterprise apps, Vendure plugins	Multi-runtime APIs, edge functions, microservices	Bun-native high-performance APIs

Hono: The Framework That Runs Everywhere

Hono is a web framework built for the edge. It runs on Node.js, Bun, Deno, Cloudflare Workers, AWS Lambda, Vercel Edge, and Fastly. The same code deploys to any runtime without changes.

import { Hono } from 'hono';
import { cors } from 'hono/cors';
import { jwt } from 'hono/jwt';
import { logger } from 'hono/logger';

const app = new Hono();

// Middleware
app.use('*', logger());
app.use('*', cors({ origin: ['https://app.example.com'] }));
app.use('/api/*', jwt({ secret: process.env.JWT_SECRET! }));

// Routes
app.get('/api/products', async (c) => {
    const products = await productService.findAll();
    return c.json({ data: products });
});

app.get('/api/products/:id', async (c) => {
    const product = await productService.findById(c.req.param('id'));
    if (!product) return c.json({ error: 'Not found' }, 404);
    return c.json({ data: product });
});

app.post('/api/products', async (c) => {
    const body = await c.req.json();
    const product = await productService.create(body);
    return c.json({ data: product }, 201);
});

export default app;

Why We Like It

Runtime portability. We deploy the same Hono codebase to Lambda (API Gateway), Bun (container), and Cloudflare Workers (edge). No code changes between runtimes. This is Hono's unique strength. No other framework can do this.

14KB core. Cold starts on Lambda are under 100ms. NestJS cold starts are 2-5 seconds. For serverless deployments, this difference is the entire user experience.

No opinions on architecture. Hono gives you routing, middleware, and context. You decide how to structure services, repositories, and business logic. For small to medium APIs (5-50 endpoints), this is exactly right. You don't need a module system for 20 routes.

tRPC integration. We use Hono + tRPC for TypeScript-only stacks. End-to-end type safety from server to client without GraphQL:

import { trpcServer } from '@hono/trpc-server';
import { appRouter } from './trpc/router';

app.use('/trpc/*', trpcServer({ router: appRouter }));

Where It Falls Short

No built-in DI. For large applications with 50+ services and complex dependency graphs, you need to bring your own DI container (tsyringe, inversify) or wire dependencies manually. This is fine for small services. It becomes tedious for large ones.

Smaller middleware ecosystem. NestJS has passport, class-validator, class-transformer, swagger, and hundreds of community packages. Hono's ecosystem is growing but not there yet. You'll write more custom middleware.

No conventions for large apps. NestJS modules give you a structure that scales to 100+ files. Hono doesn't prescribe structure. For teams with junior developers, the lack of conventions can lead to inconsistent codebases.

ElysiaJS: When Bun Performance Matters

ElysiaJS is built specifically for Bun. It takes advantage of Bun's native APIs, FFI bindings, and optimized HTTP server to achieve performance that Node.js frameworks can't match.

import { Elysia, t } from 'elysia';
import { swagger } from '@elysiajs/swagger';
import { jwt } from '@elysiajs/jwt';

const app = new Elysia()
    .use(swagger())
    .use(jwt({ name: 'jwt', secret: process.env.JWT_SECRET! }))
    .get('/products', async () => {
        return productService.findAll();
    })
    .post('/products', async ({ body }) => {
        return productService.create(body);
    }, {
        body: t.Object({
            name: t.String(),
            price: t.Number({ minimum: 0 }),
            description: t.Optional(t.String()),
        }),
    })
    .listen(3000);

Why It's Interesting

Built-in validation with TypeBox. Schemas defined inline, automatically generate OpenAPI docs. No separate validation library needed. The types flow from schema to handler to response.

Bun-native performance. ElysiaJS on Bun handles 2-3x more requests per second than Express on Node.js for simple JSON APIs. For compute-light, I/O-heavy workloads (most APIs), the difference is less dramatic but still measurable.

End-to-end type inference. The request body type is inferred from the TypeBox schema. The response type is inferred from the handler. The client can use Eden (ElysiaJS's tRPC-like client) for type-safe API calls.

Where It Falls Short

Bun-only. If you need to deploy to Node.js, Lambda, or Cloudflare Workers, ElysiaJS is not an option. Bun is production-ready but not universally supported in all hosting environments.

Smaller ecosystem. Bun's package compatibility is good but not 100%. Some Node.js packages use native addons that Bun doesn't support. Database drivers, image processing libraries, and native crypto modules can be problematic.

Less battle-tested. Bun and ElysiaJS are newer. The community is smaller. When you hit an edge case, there are fewer Stack Overflow answers and fewer people who've solved the same problem.

Drizzle vs Prisma vs TypeORM: The ORM Reality

The framework choice is half the decision. The ORM choice is the other half.

Feature	Drizzle	Prisma	TypeORM
Query style	SQL-like (explicit)	Client API (generated)	Active Record or Data Mapper
Type safety	Excellent (inferred from schema)	Excellent (generated client)	Good (decorators)
Migration	SQL-based, manual control	Auto-generated, managed	Auto-generated or manual
Performance	Fast (thin wrapper over SQL)	Good (query engine)	Moderate (heavy abstraction)
Raw SQL	First-class (sql`` template)	Possible but awkward	Possible via query builder
Edge/Serverless	Works (lightweight)	Needs Prisma Accelerate for edge	Too heavy for edge
Learning curve	Low (if you know SQL)	Low (API docs are excellent)	Medium (decorators, relations)
Bundle size	Small (~500KB)	Large (~15MB with engine)	Medium (~5MB)
Best with	Hono, ElysiaJS	Any framework	NestJS, Vendure

Our Stack Choices

Hono + Drizzle + tRPC: For new TypeScript APIs. Lightweight, fast, type-safe end-to-end. Drizzle's SQL-like syntax gives us explicit control over queries without the abstraction overhead of TypeORM or the engine overhead of Prisma.

`typescript // Drizzle: SQL-like, explicit, type-safe const products = await db .select() .from(productsTable) .where(and( eq(productsTable.tenantId, tenantId), eq(productsTable.status, 'active'), gt(productsTable.price, 1000), )) .orderBy(desc(productsTable.createdAt)) .limit(20); `

NestJS + TypeORM: For Vendure plugins and large enterprise applications where the module system and DI container justify their weight. TypeORM integrates deeply with NestJS and Vendure.

Prisma: When we need auto-generated migrations and the team values Prisma's documentation and developer experience over raw query control.

Deployment Flexibility

This is where the framework choice has the biggest practical impact.

Deployment Target	NestJS	Hono	ElysiaJS
Docker/Kubernetes	Works	Works	Works (Bun image)
AWS Lambda	Slow cold starts (2-5s)	Fast (< 100ms)	Not supported
Cloudflare Workers	Not supported	Works natively	Not supported
Vercel Edge	Not supported	Works natively	Not supported
Deno Deploy	Not supported	Works natively	Not supported
Traditional VPS	Works	Works	Works (needs Bun)
Bun native	Partial (most NestJS works on Bun)	Works	Native, optimized

If you deploy to containers only, all three work fine. If you deploy to serverless or edge, Hono is the only framework that gives you every option. ElysiaJS is Bun-only. NestJS is Node-only (practically).

For how we deploy services to cloud infrastructure including Kubernetes, Lambda, and edge functions, that page covers our approach.

When NestJS Is Still the Right Choice

Despite our preference for Hono on new projects, NestJS is the right choice when:

You're building Vendure plugins. Vendure IS NestJS. The plugin system, services, resolvers, and guards all use NestJS patterns. Fighting the framework is worse than using it.
Your team knows Angular. NestJS's module/provider/controller pattern mirrors Angular. If your team comes from Angular, the learning curve is near zero.
You need a large middleware ecosystem. Passport strategies, class-validator, Swagger auto-generation, GraphQL code-first. NestJS has it all built or well-integrated.
The application will grow to 100+ files. NestJS modules provide structure that scales. Without conventions (Hono, ElysiaJS), large teams need discipline to maintain consistency.
You need WebSocket support. NestJS has built-in WebSocket gateways with the same decorator pattern. Hono's WebSocket support is more manual.

Common Pitfalls

Choosing Hono and then rebuilding NestJS. If you end up adding a DI container, decorator-based routing, module system, and guard pattern to Hono, you've just built a worse NestJS. Use NestJS.
Choosing ElysiaJS for production before evaluating Bun compatibility. Check that all your dependencies work on Bun. Native addons, specific database drivers, and some crypto libraries may not.
Ignoring cold start times on serverless. NestJS on Lambda is unusable for user-facing APIs without provisioned concurrency. Hono on Lambda is fast enough without it.
Using TypeORM with Hono. TypeORM is designed for NestJS's DI patterns. Without DI, managing TypeORM connections and repositories is awkward. Use Drizzle or Prisma with Hono.
Premature framework switching. If your NestJS app works and your team is productive, don't switch to Hono for the sake of being modern. Switch when you have a concrete reason (serverless deployment, edge functions, new standalone service).
No validation layer. Hono doesn't include request validation. Use Zod middleware or build your own. Shipping an API without input validation is a security vulnerability.

Key Takeaways

Hono is the most versatile TypeScript backend framework. 14KB core, runs on every runtime, fast cold starts, no opinions on architecture. Our default for new standalone APIs.
ElysiaJS delivers the best performance on Bun. Built-in validation, great DX, type-safe end-to-end. But Bun-only limits where you can deploy it.
NestJS is still the right choice for large enterprise apps and Vendure. Modules, DI, guards, and a massive ecosystem. The learning curve pays off at scale.
Drizzle is our preferred ORM with Hono. SQL-like syntax, explicit queries, lightweight bundle. Prisma for teams that prefer auto-generated clients. TypeORM for NestJS/Vendure.
The deployment target drives the framework choice. Serverless and edge need Hono. Containers work with anything. Bun-native needs ElysiaJS.
Don't switch frameworks without a reason. If NestJS is working, keep using it. Switch when you need serverless, edge, or a new standalone service where the lightweight approach is genuinely better.

OpenTelemetry in Production: Traces, Context, and What Actually Matters

refaat Al Ktifan — Tue, 28 Apr 2026 17:48:50 +0000

Why OpenTelemetry Won

Three years ago, the observability landscape was fragmented. Jaeger for tracing, Prometheus for metrics, Fluentd for logs, each with its own SDK, its own protocol, its own vendor lock-in. OpenTelemetry unified them into a single standard: one SDK, one protocol (OTLP), one collector that routes to any backend.

We adopted OpenTelemetry across our production systems. This article covers the patterns that actually matter in production, not the setup tutorial. For the broader observability strategy (when to alert, what to log, how to structure metrics), see our AI observability guide. This article goes deeper on OpenTelemetry-specific implementation.

Context Propagation: The Hardest Part

A single user request might cross 5 services, 3 message queues, 2 worker processes, and an LLM call. Context propagation ensures that the trace follows the request across all of them.

HTTP Propagation (Easy)

OpenTelemetry auto-instruments HTTP clients and servers. The trace context propagates via traceparent and tracestate headers. This works out of the box.

// Auto-instrumented: no code needed for HTTP propagation
// The SDK adds traceparent header to outgoing requests
// The receiving service extracts it and continues the trace

Queue Propagation (Hard)

Message queues break automatic propagation. When you enqueue a message, the trace context must be serialized into the message headers. When a worker dequeues it, the context must be extracted and the trace continued.

// Producer: inject trace context into message headers
import { context, propagation } from '@opentelemetry/api';

async function enqueueMessage(queue: string, payload: any) {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);

    await messageQueue.send(queue, {
        body: payload,
        headers: carrier, // Contains traceparent, tracestate
    });
}

// Consumer: extract trace context from message headers
async function processMessage(message: QueueMessage) {
    const parentContext = propagation.extract(context.active(), message.headers);

    await context.with(parentContext, async () => {
        const span = tracer.startSpan('process_message', {
            attributes: {
                'messaging.system': 'rabbitmq',
                'messaging.operation': 'process',
                'messaging.destination': message.queue,
            },
        });

        try {
            await handleMessage(message.body);
            span.setStatus({ code: SpanStatusCode.OK });
        } catch (error) {
            span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
            throw error;
        } finally {
            span.end();
        }
    });
}

This pattern works for RabbitMQ, Kafka, BullMQ, SQS, and Symfony Messenger. The message headers carry the trace context. The consumer extracts it and creates child spans under the original trace.

Worker Process Propagation

For Vendure's BullMQ workers, Pimcore's Symfony Messenger workers, and similar background job systems, the pattern is the same: serialize context into the job payload, extract on the worker side.

// BullMQ: add trace context to job data
async function addJob(queue: Queue, data: any) {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);

    await queue.add('process', {
        ...data,
        _traceContext: carrier,
    });
}

// BullMQ: extract trace context in worker
worker.on('process', async (job) => {
    const parentContext = propagation.extract(context.active(), job.data._traceContext || {});

    await context.with(parentContext, async () => {
        const span = tracer.startSpan(`job:${job.name}`);
        try {
            await processJob(job.data);
        } finally {
            span.end();
        }
    });
});

Tracing LLM Calls

LLM calls are the most expensive operations in AI systems. Tracing them with proper attributes enables cost tracking, latency analysis, and quality monitoring.

async function tracedLlmCall(prompt: string, options: LlmOptions): Promise<string> {
    const span = tracer.startSpan('llm.generate', {
        attributes: {
            'llm.provider': options.provider,           // "openai", "anthropic"
            'llm.model': options.model,                 // "gpt-4o", "claude-sonnet-4-20250514"
            'llm.temperature': options.temperature,
            'llm.max_tokens': options.maxTokens,
            'llm.prompt_tokens': estimateTokens(prompt), // estimate before call
        },
    });

    try {
        const response = await llmClient.generate(prompt, options);

        span.setAttributes({
            'llm.response_tokens': response.usage.completionTokens,
            'llm.total_tokens': response.usage.totalTokens,
            'llm.finish_reason': response.finishReason,
            'llm.cost_usd': calculateCost(response.usage, options.model),
        });

        span.setStatus({ code: SpanStatusCode.OK });
        return response.text;
    } catch (error) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
        span.setAttribute('llm.error_type', error.constructor.name);
        throw error;
    } finally {
        span.end();
    }
}

Do NOT put prompt text in span attributes. Prompts contain PII. Span attributes are sent to your observability backend (Jaeger, Grafana Tempo, Datadog). Instead, log the prompt hash or token count. For the full PII-safe logging architecture, see our AI data leakage guide.

LLM Span Attributes Convention

Attribute	Type	Example
`llm.provider`	string	"openai"
`llm.model`	string	"gpt-4o"
`llm.temperature`	float	0.7
`llm.prompt_tokens`	int	1250
`llm.response_tokens`	int	340
`llm.total_tokens`	int	1590
`llm.finish_reason`	string	"stop"
`llm.cost_usd`	float	0.023
`llm.error_type`	string	"RateLimitError"
`llm.cache_hit`	boolean	false

Sampling Strategies

In high-volume AI systems, tracing every request is too expensive. Sampling reduces volume while preserving visibility into important traces.

Head-Based Sampling

Decide at the start of the trace whether to sample it. Simple but lossy.

// Sample 10% of all traces
const sampler = new TraceIdRatioBased(0.1);

// Always sample errors (override ratio for error traces)
const compositeSampler = new ParentBasedSampler({
    root: new TraceIdRatioBased(0.1),
    // Errors always sampled via span processor
});

Tail-Based Sampling (Recommended for AI)

Decide after the trace completes whether to keep it. Keeps all interesting traces (errors, slow responses, high cost) and drops routine ones.

// OpenTelemetry Collector: tail-based sampling config
processors:
    tail_sampling:
        decision_wait: 10s
        policies:
            # Keep all errors
            - name: errors
              type: status_code
              status_code: { status_codes: [ERROR] }

            # Keep slow traces (> 5 seconds)
            - name: slow
              type: latency
              latency: { threshold_ms: 5000 }

            # Keep expensive LLM calls (> $0.10)
            - name: expensive_llm
              type: string_attribute
              string_attribute:
                  key: llm.cost_usd
                  values: []  # Custom: filter in pipeline
                  enabled_regex_matching: true

            # Sample 5% of everything else
            - name: baseline
              type: probabilistic
              probabilistic: { sampling_percentage: 5 }

Tail-based sampling requires the OpenTelemetry Collector. The collector buffers complete traces, evaluates policies, and forwards only sampled traces to the backend. This adds latency (the decision_wait period) but dramatically reduces storage costs while keeping all interesting data.

Privacy-Safe Spans

Span attributes, span names, and span events are all sent to your observability backend. If any of these contain PII, your tracing infrastructure becomes a data protection liability.

// BAD: PII in span attributes
span.setAttribute('user.email', 'sara.mustermann@beispiel.de');
span.setAttribute('user.name', 'Sara Mustermann');
span.setAttribute('request.body', JSON.stringify(requestBody)); // Contains PII

// GOOD: token IDs and aggregate data only
span.setAttribute('user.id', 'usr_abc123');  // Opaque ID, not PII
span.setAttribute('entities.detected', 3);
span.setAttribute('entities.types', ['person', 'email', 'phone']);
span.setAttribute('policy.applied', 'german-support');

Rules for privacy-safe tracing:

User IDs: opaque identifiers only (not emails, not names)
Request bodies: never include raw content. Log entity counts and types.
LLM prompts: never include. Log token counts and prompt hash.
Error messages: sanitize before attaching to spans. Strip any user data.

The Baggage API

OpenTelemetry Baggage carries key-value pairs across service boundaries. Unlike span attributes (which stay on the span), baggage propagates to all downstream services automatically.

import { propagation, context, baggage } from '@opentelemetry/api';

// Set baggage at the API gateway
const bag = propagation.createBaggage({
    'tenant.id': { value: 'tenant_acme' },
    'request.priority': { value: 'high' },
    'feature.flags': { value: 'new-checkout,beta-search' },
});
const ctx = propagation.setBaggage(context.active(), bag);

// Downstream services can read baggage
const tenantId = propagation.getBaggage(context.active())?.getEntry('tenant.id')?.value;

Useful for:

Tenant ID propagation (every downstream service knows which tenant)
Feature flags (propagate experiment assignments across services)
Priority routing (high-priority requests get different queue treatment)
Debug markers (mark specific requests for verbose logging)

Baggage travels with the trace context in HTTP headers and message metadata. Every service that extracts the trace context also gets the baggage.

Collector Architecture

The OpenTelemetry Collector is the central routing layer between your applications and your observability backends.

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Service A   │  │  Service B   │  │  Worker C    │
│  (OTLP gRPC) │  │  (OTLP HTTP) │  │  (OTLP gRPC) │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       ▼                 ▼                 ▼
┌─────────────────────────────────────────────────┐
│              OTel Collector                      │
│                                                  │
│  Receivers: OTLP (gRPC + HTTP)                  │
│  Processors: batch, tail_sampling, attributes    │
│  Exporters: Tempo, Prometheus, Loki             │
└─────────────────────────────────────────────────┘
       │                 │                 │
       ▼                 ▼                 ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│  Grafana     │ │  Prometheus  │ │  Grafana     │
│  Tempo       │ │              │ │  Loki        │
│  (traces)    │ │  (metrics)   │ │  (logs)      │
└──────────────┘ └──────────────┘ └──────────────┘

The collector handles batching (reduces network calls), sampling (reduces storage), attribute processing (adds/removes attributes), and routing (different signals to different backends). Deploy it as a sidecar or as a central service depending on your infrastructure.

For cloud deployment patterns including observability infrastructure, that page covers our approach.

Common Pitfalls

No context propagation across queues. HTTP propagation is automatic. Queue propagation is not. If you don't inject/extract context in message headers, traces break at every queue boundary.
PII in span attributes. Your tracing backend indexes everything. If spans contain emails, names, or request bodies, your Grafana Tempo cluster is a PII store.
Tracing every request in production. At 1000 RPS, full tracing generates terabytes of data. Use tail-based sampling to keep errors, slow traces, and expensive operations.
No LLM-specific attributes. Without token counts, cost, model ID, and finish reason on LLM spans, you can't track AI costs or diagnose quality issues.
Head-based sampling dropping errors. If you sample 10% of traces and an error happens in the 90% you drop, you never see it. Use tail-based sampling or always-sample-errors policies.
Baggage for large payloads. Baggage travels with every request. Large values increase header size on every HTTP call. Keep baggage values small (IDs, flags, priorities).

Key Takeaways

Context propagation across queues is the hardest part. HTTP is automatic. Queues require manual inject/extract of trace context in message headers. This is where most distributed tracing implementations break.
Trace LLM calls with cost and token attributes. Model, provider, token counts, cost, finish reason. These attributes enable AI cost dashboards and quality monitoring.
Tail-based sampling for AI workloads. Keep all errors, slow traces, and expensive operations. Drop routine traces. Reduces storage by 90%+ while keeping all interesting data.
No PII in spans. Opaque user IDs, entity counts, token types. Never raw content, emails, names, or request bodies.
Baggage propagates tenant context. Set tenant ID, feature flags, and priority at the edge. Every downstream service reads it from baggage without explicit parameter passing.

Event-Driven Architecture in Practice: What Actually Goes Wrong

refaat Al Ktifan — Sat, 25 Apr 2026 15:26:51 +0000

The Event Storm Nobody Expects

Event-driven architecture looks clean on a whiteboard. Service A emits an event. Service B consumes it. Loose coupling. Independent scaling. Beautiful diagrams.

Then you deploy to production. One product save triggers 8 event subscribers. Each subscriber dispatches 3 async messages. Each message handler loads the product, modifies a field, and saves. Each save triggers 8 subscribers again. Within seconds, your message queue has thousands of messages for a single product update, your workers are saturated, and your database is under lock contention.

We've operated four different event-driven systems in production. Each used a different message broker. Each had different failure patterns. This article covers what actually goes wrong and how to fix it. For broader context on how we approach system architecture, that guide covers our methodology.

The Four Failure Patterns

Every event-driven system eventually encounters these:

Pattern	What Happens	Severity
Event storm	One action cascades into thousands of messages	Critical (can saturate infrastructure)
Bidirectional sync loop	System A syncs to B, B syncs back to A, infinite loop	Critical (infinite message growth)
Duplicate processing	Same message processed twice, creates duplicate data	High (data integrity)
Dead letter accumulation	Failed messages pile up with no resolution strategy	Medium (operational debt)

1. Event Storms

The most common and most dangerous pattern. It happens when event handlers trigger new events, and those events trigger more handlers.

Product save
  -> EventSubscriber A dispatches MessageA
  -> EventSubscriber B dispatches MessageB
  -> EventSubscriber C dispatches MessageC
  -> ...8 subscribers total

WorkerA processes MessageA
  -> Modifies product, calls save()
  -> Product save triggers 8 subscribers again
  -> Each dispatches more messages

WorkerB processes MessageB (same pattern)
  -> 8 more messages

Result: 1 save -> 8 messages -> 8 saves -> 64 messages -> ...

The solution: EventSubscriberSupervisor. A process-scoped control that disables event subscribers during worker saves.

// Worker handler: disable subscribers before saving
async function handleGenerateThumbnail(message: ThumbnailMessage) {
    const product = await productService.findById(message.productId);

    subscriberSupervisor.disableAll();
    try {
        product.thumbnail = await generateThumbnail(product);
        await product.save(); // No subscribers fire, no new messages dispatched
    } finally {
        subscriberSupervisor.enableAll();
    }
}

The order matters. The save must happen inside the disabled scope. If you re-enable subscribers before saving, the save triggers the cascade.

For a deeper look at how we apply this pattern specifically in Pimcore, see our Pimcore workflow design guide.

2. Bidirectional Sync Loops

When two systems sync data bidirectionally, each update from A triggers a sync to B, which triggers a sync back to A. Without loop prevention, this runs forever.

System A (Commerce)         System B (Operations)
     │                           │
     │  Customer updated         │
     │ ────────────────────────▶ │
     │                           │  Sync received, save customer
     │                           │  Customer updated event fires
     │  Sync received            │
     │ ◀──────────────────────── │
     │  Save customer            │
     │  Customer updated event   │
     │ ────────────────────────▶ │
     │           ...forever...   │

The solution: source tracking. Every message carries an x-source header that identifies the originating system. When a system receives a message from itself (via the other system), it ignores it.

// When publishing a sync message
async function publishCustomerSync(customer: Customer, source: string) {
    await messageQueue.publish('customer.updated', {
        customerId: customer.id,
        data: customer,
        source: source,  // "commerce" or "operations"
    });
}

// When consuming a sync message
async function handleCustomerSync(message: CustomerSyncMessage) {
    // Ignore messages that originated from this system
    if (message.source === THIS_SYSTEM_ID) {
        logger.debug('Ignoring self-originated sync', { customerId: message.customerId });
        return; // ACK the message, don't process
    }

    const customer = await customerService.update(message.customerId, message.data);

    // When saving, mark the source so downstream events carry it
    await publishCustomerSync(customer, THIS_SYSTEM_ID);
}

An alternative approach: message deduplication by content hash. Hash the message payload and check if the same hash was processed recently. If the data hasn't changed, the sync is a no-op.

3. Duplicate Processing

Network failures, worker restarts, and at-least-once delivery guarantees mean the same message can be delivered more than once. Without idempotency, you get duplicate orders, duplicate emails, duplicate database records.

The solution: idempotency stores with business-key deduplication.

// Idempotency store with business key
class IdempotencyStore {
    async checkAndAcquire(key: string, scope: string): Promise<boolean> {
        try {
            await this.db.insert('idempotency_keys', {
                key,
                scope,
                status: 'PROCESSING',
                created_at: new Date(),
                expires_at: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24h TTL
            });
            return true; // Acquired, proceed with processing
        } catch (error) {
            if (isDuplicateKeyError(error)) {
                return false; // Already processed, skip
            }
            throw error;
        }
    }

    async complete(key: string, scope: string): Promise<void> {
        await this.db.update('idempotency_keys',
            { status: 'COMPLETED', completed_at: new Date() },
            { key, scope }
        );
    }
}

// Usage in a worker
async function handleNotification(message: NotificationMessage) {
    const dedupeKey = `${message.recipientEmail}:${message.category}:${message.entityRef}:${todayBucket()}`;

    const acquired = await idempotencyStore.checkAndAcquire(dedupeKey, 'notification');
    if (!acquired) {
        logger.info('Duplicate notification skipped', { dedupeKey });
        return; // ACK the message
    }

    try {
        await emailService.send(message);
        await idempotencyStore.complete(dedupeKey, 'notification');
    } catch (error) {
        await idempotencyStore.fail(dedupeKey, 'notification');
        throw error; // NACK, let the queue retry
    }
}

The dedupe key must be business-meaningful. For notifications: recipient + category + entity + day bucket (prevents sending the same notification twice in one day but allows it the next day). For orders: tenant + product + date. For imports: source record ID + import batch ID.

4. Dead Letter Handling

Messages that fail repeatedly end up in a dead letter queue. Most teams set up the DLQ and then ignore it. Dead letters accumulate. Months later, someone discovers thousands of unprocessed messages.

// Dead letter handler with classification
async function processDeadLetter(message: DeadLetterMessage) {
    const failureType = classifyFailure(message.error);

    switch (failureType) {
        case 'TRANSIENT':
            // Network timeout, service temporarily down
            // Requeue with exponential backoff
            await requeue(message, { delay: calculateBackoff(message.retryCount) });
            break;

        case 'PERMANENT':
            // Invalid data, schema mismatch, business rule violation
            // Log, alert, archive. Do not retry.
            await archiveDeadLetter(message);
            await alertOps('Permanent failure in dead letter', message);
            break;

        case 'POISON':
            // Message itself causes crashes (malformed payload)
            // Archive immediately, never retry
            await archiveDeadLetter(message);
            await alertOps('Poison message detected', message);
            break;
    }
}

function classifyFailure(error: string): 'TRANSIENT' | 'PERMANENT' | 'POISON' {
    if (error.includes('ECONNREFUSED') || error.includes('timeout')) return 'TRANSIENT';
    if (error.includes('SyntaxError') || error.includes('Cannot parse')) return 'POISON';
    return 'PERMANENT';
}

The key insight: not all dead letters are the same. Transient failures should be retried. Permanent failures should be archived and investigated. Poison messages should be quarantined immediately.

For how we handle failure patterns in AI systems specifically, see our AI failure modes guide.

Choosing a Message Broker

We've used four different brokers in production. Each has a distinct sweet spot.

Broker	Best For	Not Great For	Deployment Complexity
Kafka	High-throughput event streaming, event sourcing, replay	Simple job queues, low-volume systems	High (ZooKeeper/KRaft, partitions, consumer groups)
RabbitMQ	Routing, dead letters, priority queues, complex topologies	Very high throughput (millions/sec), event replay	Medium (clustering, HA policies)
BullMQ (Redis)	Job queues, delayed jobs, rate limiting, simple setups	Complex routing, message replay, multi-consumer patterns	Low (just Redis)
Symfony Messenger	PHP/Symfony apps, Pimcore, simple async dispatch	Non-PHP ecosystems, complex broker features	Low (uses Doctrine, AMQP, or Redis as transport)

When to Use Kafka

You need event replay (consumers can re-read from any offset)
You have multiple consumer groups processing the same events differently
Throughput exceeds 100K messages per second
You're doing event sourcing or building a change data capture pipeline
You need guaranteed ordering within a partition

When to Use RabbitMQ

You need complex routing (topic exchanges, headers-based routing)
Dead letter queues with automatic retry policies are important
Message priority matters (urgent messages processed first)
You need request-reply patterns (RPC over messages)
Your system has multiple services in different languages

When to Use BullMQ

You're in a Node.js/TypeScript ecosystem (Vendure, NestJS)
Job scheduling with delays and cron patterns
Rate limiting per queue or per job type
You already have Redis for caching and sessions
Your system is small to medium scale (under 10K messages/sec)

When to Use Symfony Messenger

You're in a PHP/Symfony/Pimcore ecosystem
You need simple async dispatch for background tasks
Worker infrastructure follows Symfony conventions (supervisord)
Transport flexibility (switch between Doctrine, AMQP, Redis without code changes)

For our Pimcore projects, we use Symfony Messenger with RabbitMQ transport. For Vendure projects, we use BullMQ. For high-throughput data ingestion platforms, we use Kafka. The broker choice follows the ecosystem, not the other way around.

Message Ordering

Most teams assume message ordering matters. Usually it doesn't.

Scenario	Ordering Needed?	Why
Send notification email	No	Emails are inherently unordered
Generate thumbnail	No	Latest version wins
Update search index	No	Latest state wins (eventual consistency)
Process payment steps	Yes	Charge must happen before capture
Order state transitions	Yes	Can't ship before payment confirmed
Event sourcing replay	Yes	Events must replay in causal order

When ordering matters, use FIFO queues (SQS FIFO, Kafka partitions keyed by entity ID, RabbitMQ with single consumer). When it doesn't, parallel processing is faster and simpler.

// Kafka: ordering within a partition (keyed by product ID)
await producer.send({
    topic: 'product-updates',
    messages: [{
        key: productId,  // All updates for same product go to same partition
        value: JSON.stringify(event),
    }],
});

// BullMQ: no ordering needed for thumbnails
await thumbnailQueue.add('generate', { productId }, {
    removeOnComplete: true,
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
});

Retry Strategies

Not all retries are equal. The strategy depends on the failure type.

Strategy	When to Use	Example
Immediate retry	Transient glitch (race condition)	Optimistic lock failure
Exponential backoff	Service temporarily down	External API timeout
Fixed delay	Rate limiting	API returns 429
No retry	Permanent failure	Invalid payload, business rule violation

// BullMQ retry configuration
const queue = new Queue('notifications', {
    defaultJobOptions: {
        attempts: 5,
        backoff: {
            type: 'exponential',
            delay: 60000, // 1min, 2min, 4min, 8min, 16min
        },
        removeOnComplete: { count: 1000 },
        removeOnFail: false, // Keep for dead letter analysis
    },
});

After all retries are exhausted, the message moves to a dead letter queue. Never retry infinitely. Set a maximum attempt count and handle the failure explicitly.

Monitoring Event-Driven Systems

The hardest part of event-driven systems is debugging. When something fails, the error might be 5 services and 12 messages away from the original cause.

Correlation IDs

Every message carries a correlation ID that links it to the original action:

// Original API request generates correlation ID
const correlationId = generateUUID();

// Every message in the chain carries it
await queue.add('process-order', {
    orderId: order.id,
    correlationId,
});

// Workers pass it to downstream messages
async function handleProcessOrder(job) {
    const { orderId, correlationId } = job.data;

    // ... process order ...

    // Downstream messages carry the same correlation ID
    await notificationQueue.add('send-confirmation', {
        orderId,
        correlationId, // Same ID, traceable back to the original request
    });
}

When debugging, query all logs and events by correlation ID to see the full chain. For more on distributed tracing patterns, see our AI observability guide.

Queue Health Metrics

Metric	Healthy	Warning	Critical
Queue depth	< 100	100-1000	> 1000
Processing time (p95)	< 5s	5-30s	> 30s
Failure rate	< 1%	1-5%	> 5%
Dead letter count	0	1-10	> 10
Consumer lag (Kafka)	< 1000	1K-10K	> 10K

Alert on queue depth trends, not absolute values. A queue depth of 500 that's been stable for hours is fine. A queue depth of 50 that was 0 an hour ago and is growing is a problem.

Common Pitfalls

No subscriber control during worker saves. The single biggest source of event storms. Worker saves must not trigger the same event subscribers that dispatch work to workers.
No source tracking on bidirectional sync. Without x-source headers, bidirectional sync between two systems loops forever.
Using message ID for idempotency. Message IDs change on redelivery in some brokers. Use business keys (entity ID + action + time bucket) instead.
Infinite retries. Set a maximum retry count. After exhaustion, move to dead letter queue. Never retry forever.
Ignoring dead letter queues. Dead letters are operational debt. Classify them (transient, permanent, poison) and handle each type differently.
Assuming message ordering. Most workloads don't need ordering. Parallel processing is faster. Only use FIFO when state transitions or causal ordering require it.
Same retry strategy for all failures. A timeout needs exponential backoff. An invalid payload needs zero retries. A rate limit needs fixed delay.
No correlation IDs. Without them, debugging a failure chain across 5 services is impossible.

Key Takeaways

Event storms are the most dangerous failure mode. One uncontrolled save cascade can saturate your entire infrastructure. The EventSubscriberSupervisor is not optional.
Bidirectional sync needs source tracking. Every message carries the originating system ID. When you receive a message from yourself (via the other system), ignore it.
Idempotency uses business keys, not message IDs. Recipient + category + entity + day bucket for notifications. Tenant + product + date for orders. The key must be business-meaningful.
Dead letters need classification. Transient failures get retried. Permanent failures get archived. Poison messages get quarantined. Never ignore the dead letter queue.
The broker follows the ecosystem. Kafka for high-throughput streaming, RabbitMQ for complex routing, BullMQ for Node.js job queues, Symfony Messenger for PHP. Don't pick a broker and then build around it.
Most workloads don't need ordering. Parallel processing is simpler and faster. Use FIFO only when state transitions demand it.

AI Browser Automation Without BrowserBase: What We Built Instead

refaat Al Ktifan — Sat, 25 Apr 2026 15:17:00 +0000

The Paid Browser Automation Market

BrowserBase, Browserless, and similar services charge per-minute or per-session for managed headless browsers. For AI workflows that need to interact with web pages (filling forms, extracting structured data, navigating multi-step processes), these services handle the infrastructure: browser instances, anti-detection, proxies, and session management.

The pricing adds up fast. At $0.10-0.50 per session-minute, a workflow that processes 1,000 pages per day at 2 minutes each costs $200-1,000 per day. For an AI system that runs continuously, that's $6,000-30,000 per month just for browser infrastructure.

We built a self-hosted alternative using Playwright + LLM for page understanding. It handles 90% of the use cases at a fraction of the cost. This article covers the architecture. For how we build AI workflow systems and agentic AI more broadly, those guides cover the higher-level patterns.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                  AI Browser Engine                       │
│                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │  Task Queue   │  │  Instance    │  │  Session      │  │
│  │  (BullMQ)     │  │  Pool        │  │  Manager      │  │
│  │               │  │  (Playwright │  │  (cookies,    │  │
│  │  Prioritized  │  │   browsers)  │  │   localStorage│  │
│  │  Retry logic  │  │              │  │   auth state) │  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  │
│         │                 │                  │           │
│         ▼                 ▼                  ▼           │
│  ┌──────────────────────────────────────────────────┐   │
│  │              Page Interaction Layer                │   │
│  │                                                    │   │
│  │  1. Navigate to URL                               │   │
│  │  2. Wait for page load                            │   │
│  │  3. Extract page structure (accessibility tree)   │   │
│  │  4. Send structure to LLM for understanding       │   │
│  │  5. LLM returns action plan (click, type, select) │   │
│  │  6. Execute actions via Playwright                │   │
│  │  7. Extract structured data from result           │   │
│  └──────────────────────────────────────────────────┘   │
│                                                          │
└─────────────────────────────────────────────────────────┘

Instance Pooling

Running a new browser for every task is expensive (cold start: 1-3 seconds, memory: 200-400MB per instance). A pool reuses browser instances across tasks.

class BrowserPool {
    private available: Browser[] = [];
    private inUse = new Map<string, Browser>();
    private maxInstances: number;

    constructor(options: { maxInstances: number }) {
        this.maxInstances = options.maxInstances;
    }

    async acquire(): Promise<{ browser: Browser; id: string }> {
        // Reuse an available instance
        if (this.available.length > 0) {
            const browser = this.available.pop()!;
            const id = crypto.randomUUID();
            this.inUse.set(id, browser);
            return { browser, id };
        }

        // Create new if under limit
        if (this.inUse.size < this.maxInstances) {
            const browser = await chromium.launch({
                headless: true,
                args: [
                    '--no-sandbox',
                    '--disable-setuid-sandbox',
                    '--disable-dev-shm-usage',
                    '--disable-gpu',
                    '--single-process',
                ],
            });
            const id = crypto.randomUUID();
            this.inUse.set(id, browser);
            return { browser, id };
        }

        // Pool exhausted: wait for one to be released
        return new Promise((resolve) => {
            this.waitQueue.push(resolve);
        });
    }

    async release(id: string): Promise<void> {
        const browser = this.inUse.get(id);
        if (!browser) return;

        this.inUse.delete(id);

        // Clear state between tasks
        const pages = browser.contexts();
        for (const context of pages) {
            await context.close();
        }

        // If someone is waiting, give them this instance
        if (this.waitQueue.length > 0) {
            const resolve = this.waitQueue.shift()!;
            const newId = crypto.randomUUID();
            this.inUse.set(newId, browser);
            resolve({ browser, id: newId });
        } else {
            this.available.push(browser);
        }
    }
}

Pool Sizing

Workload	Pool Size	Memory Required
Light (< 100 pages/hour)	2-3 instances	1-2 GB
Medium (100-500 pages/hour)	5-10 instances	3-5 GB
Heavy (500+ pages/hour)	10-20 instances	5-10 GB

Each Chromium instance uses 200-400MB of RAM. The pool size determines your throughput ceiling and memory requirements. Start small and scale based on actual load.

Session Management

Many workflows require maintaining login state across multiple page interactions. The session manager persists cookies, localStorage, and authentication tokens between tasks.

class SessionManager {
    private sessions = new Map<string, SessionState>();

    async createSession(id: string, options: SessionOptions): Promise<BrowserContext> {
        const context = await browser.newContext({
            viewport: { width: 1280, height: 720 },
            userAgent: options.userAgent || this.getRandomUserAgent(),
            locale: options.locale || 'en-US',
            timezoneId: options.timezone || 'Europe/Berlin',
        });

        // Restore previous session state if exists
        const existing = this.sessions.get(id);
        if (existing) {
            await context.addCookies(existing.cookies);
            // localStorage restored via page.evaluate after navigation
        }

        return context;
    }

    async saveSession(id: string, context: BrowserContext): Promise<void> {
        const cookies = await context.cookies();
        const pages = context.pages();
        let localStorage = {};

        if (pages.length > 0) {
            localStorage = await pages[0].evaluate(() => {
                const data: Record<string, string> = {};
                for (let i = 0; i < window.localStorage.length; i++) {
                    const key = window.localStorage.key(i);
                    if (key) data[key] = window.localStorage.getItem(key) || '';
                }
                return data;
            });
        }

        this.sessions.set(id, {
            cookies,
            localStorage,
            lastUsed: Date.now(),
        });
    }
}

LLM-Driven Page Understanding

The core innovation: instead of writing CSS selectors or XPath queries for every page, send the page's accessibility tree to an LLM and let it decide which elements to interact with.

async function extractPageStructure(page: Page): Promise<string> {
    // Get the accessibility tree (structured, compact representation)
    const tree = await page.accessibility.snapshot();

    // Convert to a text format the LLM can understand
    return formatAccessibilityTree(tree, {
        maxDepth: 5,
        includeRoles: ['button', 'link', 'textbox', 'combobox', 'checkbox', 'heading'],
        includeText: true,
        includeLabels: true,
    });
}

function formatAccessibilityTree(node: any, options: any, depth = 0): string {
    if (depth > options.maxDepth) return '';
    if (!options.includeRoles.includes(node.role) && depth > 1) {
        // Skip non-interactive elements, but recurse into children
        return (node.children || []).map(c => formatAccessibilityTree(c, options, depth + 1)).join('');
    }

    const indent = '  '.repeat(depth);
    let result = `${indent}[${node.role}] ${node.name || ''}`;
    if (node.value) result += ` value="${node.value}"`;
    result += '\n';

    for (const child of node.children || []) {
        result += formatAccessibilityTree(child, options, depth + 1);
    }
    return result;
}

LLM Action Planning

Send the page structure to the LLM with the task description. The LLM returns a sequence of actions:

async function planActions(pageStructure: string, task: string): Promise<Action[]> {
    const response = await llm.generate({
        model: 'gpt-4o-mini', // Fast model for action planning
        messages: [
            {
                role: 'system',
                content: `You are a browser automation assistant. Given a page structure and a task,
                return a JSON array of actions to accomplish the task.
                Available actions: click(selector), type(selector, text), select(selector, value),
                wait(ms), extract(selector).
                Use the element text/labels to identify targets, not CSS selectors.`,
            },
            {
                role: 'user',
                content: `Page structure:\n${pageStructure}\n\nTask: ${task}`,
            },
        ],
        responseFormat: 'json',
    });

    return JSON.parse(response.text);
}

// Example task: "Fill in the contact form with name Sara Mustermann and email sara.mustermann@beispiel.de"
// LLM returns:
// [
//   { "action": "type", "target": "Name input field", "value": "Sara Mustermann" },
//   { "action": "type", "target": "Email input field", "value": "sara.mustermann@beispiel.de" },
//   { "action": "click", "target": "Submit button" }
// ]

Resolving LLM Actions to Playwright Commands

The LLM returns human-readable targets ("Name input field"). A resolver maps them to Playwright selectors:

async function resolveAndExecute(page: Page, actions: Action[]): Promise<void> {
    for (const action of actions) {
        // Find the element matching the LLM's description
        const element = await findElementByDescription(page, action.target);

        if (!element) {
            throw new ActionError(`Could not find element: ${action.target}`);
        }

        switch (action.action) {
            case 'click':
                await element.click();
                await page.waitForLoadState('networkidle');
                break;
            case 'type':
                await element.fill(action.value);
                break;
            case 'select':
                await element.selectOption(action.value);
                break;
            case 'wait':
                await page.waitForTimeout(action.value);
                break;
            case 'extract':
                const text = await element.textContent();
                results.push({ field: action.target, value: text });
                break;
        }
    }
}

async function findElementByDescription(page: Page, description: string): Promise<ElementHandle | null> {
    // Try multiple strategies to find the element
    const strategies = [
        // By aria-label
        () => page.$(`[aria-label*="${description}" i]`),
        // By placeholder
        () => page.$(`[placeholder*="${description}" i]`),
        // By visible text
        () => page.$(`text=${description}`),
        // By label association
        () => page.$(`label:has-text("${description}") + input, label:has-text("${description}") input`),
        // By role and name
        () => page.getByRole('textbox', { name: new RegExp(description, 'i') }).first().elementHandle(),
        () => page.getByRole('button', { name: new RegExp(description, 'i') }).first().elementHandle(),
    ];

    for (const strategy of strategies) {
        try {
            const element = await strategy();
            if (element) return element;
        } catch {
            continue;
        }
    }

    return null;
}

Anti-Detection Basics

Some websites detect and block headless browsers. Basic countermeasures:

const context = await browser.newContext({
    // Randomize viewport
    viewport: {
        width: 1280 + Math.floor(Math.random() * 200),
        height: 720 + Math.floor(Math.random() * 100),
    },

    // Rotate user agents
    userAgent: getRandomUserAgent(),

    // Set realistic locale and timezone
    locale: 'de-DE',
    timezoneId: 'Europe/Berlin',

    // Realistic geolocation
    geolocation: { latitude: 48.1351, longitude: 11.5820 },
    permissions: ['geolocation'],
});

// Override navigator.webdriver (headless detection)
await page.addInitScript(() => {
    Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

Note: anti-detection is an arms race. For sites with sophisticated bot detection (Cloudflare, Akamai), self-hosted Playwright will eventually be detected. This is where paid services like BrowserBase add value: they invest continuously in anti-detection. For most business automation tasks (internal tools, partner portals, public data), basic anti-detection is sufficient.

When Paid Tools ARE Worth It

Scenario	Self-Hosted	Paid Service
Internal tool automation	Best choice (no anti-detection needed)	Overkill
Public data extraction (simple)	Good (basic anti-detection works)	Unnecessary
Sites with bot detection	Possible but constant maintenance	Worth it (they handle anti-detection)
High-volume scraping (10K+ pages/day)	Complex (proxy rotation, IP management)	Worth it (managed infrastructure)
Regulated data (GDPR, compliance)	Better (data stays on your infrastructure)	Risk (data goes through third party)
One-time migration	Good (temporary workload)	Unnecessary cost

The decision framework: if you're automating internal workflows or processing public data from sites without aggressive bot detection, self-host. If you're doing high-volume extraction from sites with Cloudflare-level protection, pay for a service that handles anti-detection as their core business.

Cost Comparison

Component	Self-Hosted (monthly)	BrowserBase (monthly)
Compute (5 instances)	$50-100 (container/VPS)	N/A
LLM calls (action planning)	$20-50 (GPT-4o-mini)	N/A
BrowserBase sessions	N/A	$500-2,000
Proxy service (if needed)	$50-200	Included
Maintenance	2-4 hours/month	None
Total (1,000 pages/day)	$120-350/month	$500-2,000/month
Total (10,000 pages/day)	$300-800/month	$3,000-10,000/month

Self-hosting is 3-10x cheaper at scale. The trade-off is maintenance time and anti-detection capability.

Common Pitfalls

No instance pooling. Launching a new browser per task wastes 1-3 seconds on cold start and 200-400MB of RAM. Pool and reuse instances.
Hardcoded CSS selectors. Pages change their DOM structure regularly. LLM-based element identification is more resilient than hardcoded selectors.
No session persistence. Multi-step workflows that require login fail when the session state is lost between steps.
Ignoring anti-detection entirely. Even basic measures (random viewport, user agent rotation, webdriver override) prevent detection on most sites.
Using a large model for action planning. GPT-4o-mini or Claude Haiku are fast enough for page understanding. A large model adds latency without better accuracy for this task.
No timeout on page loads. Some pages load indefinitely (infinite scrolling, slow third-party scripts). Set a navigation timeout and handle it.
Running in production without monitoring. Track success rate, average execution time, and error types per workflow. Alert when success rate drops.

Key Takeaways

Self-hosted Playwright + LLM handles 90% of browser automation use cases. For internal tools, partner portals, and public data without aggressive bot detection, this is the right approach.
Instance pooling is essential. Reuse browser instances across tasks. Cold starts and memory allocation are the biggest performance bottleneck.
LLM page understanding replaces brittle selectors. Send the accessibility tree to a fast model. Let it decide which elements to interact with. More resilient to page changes than hardcoded CSS selectors.
Paid services earn their cost on anti-detection. If your target sites have Cloudflare or similar protection, BrowserBase invests continuously in bypassing it. That's their core business. Don't try to compete.
Self-hosting is 3-10x cheaper at scale. But you pay in maintenance time and anti-detection limitations. Make the trade-off consciously.

FIND MORE: https://oronts.com/en/guides/browser-automation-ai-without-paid-tools

CodeRanedeer: An AI-Powered Programming Assistant

refaat Al Ktifan — Mon, 08 May 2023 19:14:30 +0000

Enhance Your Coding Experience with CodeRanedeer: An AI-Powered Programming Assistant (GPT 4 prompt)

AI Code Buddy - CodeRanedeer

Discover how CodeRanedeer, an AI-powered language model (GPT 4 prompt), can help you write, fix, review, and explain code with personalized learning experiences and interaction styles.

Introduction:

Whether you’re a beginner or an experienced programmer, having a helping hand while writing, fixing, or reviewing code can make a significant difference in your productivity. CodeRanedeer, an AI-powered programming assistant developed by Refaat Al Ktifan, is designed to cater to your coding needs with personalized learning experiences and interaction styles. This blog post will introduce CodeRanedeer, walk you through its features, and provide examples to showcase its capabilities. Check out the GitHub repository here: https://github.com/Refaat-alktifan/CodeRanedeer

Section 1:

Overview of CodeRanedeer CodeRanedeer is an AI language model that assists with various code-oriented tasks such as writing, fixing, reviewing, and explaining programming code. It can also serve as a pair programming buddy, enhancing your coding experience through collaboration. CodeRanedeer offers personalization options to cater to individual learning preferences and interaction styles.

Section 2:

Personalization Features CodeRanedeer provides personalized learning experiences and interaction styles based on user preferences. It offers the following personalization options:

Depth: Determines the level of detail in explanations and assistance, ranging from basic concepts to advanced techniques.
Interaction Styles: Specifies the AI’s role in the interaction, such as providing guidance or collaborating as a pair programming buddy.
Explanation Styles: Defines how the AI presents information, catering to different levels of programming expertise and learning preferences.

Section 3:

Commands and Configuration CodeRanedeer features a range of commands to help you perform code-oriented tasks. These include:

/write_code: Write code based on user’s specifications.
/fix_code: Identify and fix issues in the provided code.
/review_code: Review the provided code and offer suggestions for improvement.
/explain_code: Explain the provided code or concepts to the user.
/pair_program: Engage in pair programming with the user.

You can also configure CodeRanedeer according to your preferences, including depth, interaction style, explanation style, and language.

Section 4:

Example Use Cases Here are some examples showcasing CodeRanedeer’s capabilities:

Writing code: CodeRanedeer can write a Python function to calculate the factorial of a given number.
Reviewing code: CodeRanedeer can review your Python code, offering suggestions for improvement or optimization.
Fixing code: CodeRanedeer can identify issues in your JavaScript code and provide fixes, ensuring your code is working correctly.
Explaining code: CodeRanedeer can break down and explain Java code, making it easier to understand the logic and flow of the program.
Pair programming: CodeRanedeer can help you create a Python script to find the greatest common divisor (GCD) of two numbers, engaging in pair programming to enhance your coding experience.

Conclusion:

CodeRanedeer is an invaluable AI-powered programming assistant (GPT 4 prompt) that can help you with various code-oriented tasks while catering to your learning preferences and interaction styles. With CodeRanedeer, you can improve your coding skills, enhance your understanding of programming concepts, and boost your productivity. Don’t forget to visit the GitHub repository to explore CodeRanedeer further: https://github.com/Refaat-alktifan/CodeRanedeer
AI Code Buddy - CodeRanedeer

Keywords: CodeRanedeer, AI-powered, programming assistant, personalized learning, chatgpt, gpt4, ai, free ai

0–7 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 16:05:04 +0000

Pipes are an essential feature of NestJS, used for handling data transformation and validation in a clean and maintainable way. Pipes are classes that implement the PipeTransform interface, which requires a single method: transform(). This method takes an input value and returns a transformed value or throws an exception if the value is invalid.

There are two primary use cases for pipes in NestJS:

Validation: Pipes can validate incoming data, ensuring it meets specific requirements before it’s passed to a controller method. If the data is invalid, the pipe can throw an exception, which NestJS will convert into an appropriate HTTP error response.

For example, you can use the built-in ValidationPipe to automatically validate incoming data against a class-validator schema:

    import { Controller, Post, Body, UsePipes } from '@nestjs/common';
    import { ValidationPipe } from '@nestjs/common';
    import { CreateCatDto } from './dto/create-cat.dto';

    @Controller('cats')
    export class CatsController {
      @Post()
      @UsePipes(ValidationPipe)
      create(@Body() createCatDto: CreateCatDto) {
        // Implementation of the create method
      }
    }

Data Transformation: Pipes can transform incoming data, allowing you to modify or normalize it before it reaches a controller method. This is useful for cases like converting strings to numbers or formatting date strings.

For instance, you can create a custom pipe to parse a string into a number:

    import { PipeTransform, Injectable, BadRequestException } from '@nestjs/common';

    @Injectable()
    export class ParseIntPipe implements PipeTransform {
      transform(value: string) {
        const val = parseInt(value, 10);
        if (isNaN(val)) {
          throw new BadRequestException('Validation failed');
        }
        return val;
      }
    }

To use this custom pipe, apply it to a controller method with the @UsePipes() decorator:

    import { Controller, Get, Param, UsePipes } from '@nestjs/common';
    import { ParseIntPipe } from './parse-int.pipe';

    @Controller('cats')
    export class CatsController {
      @Get(':id')
      findOne(@Param('id', ParseIntPipe) id: number) {
        // Implementation of the findOne method
      }
    }

Pipes help maintain a clean and organized codebase by handling validation and transformation concerns separately from the core business logic.

Next, let’s explore the concept of interceptors in NestJS. How do interceptors work, and what are some common use cases for them in a NestJS application?

to be continued

0–6 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 16:02:35 +0000

Dependency Injection (DI) is a design pattern used in NestJS to manage dependencies between classes, promoting decoupling and making it easier to test and maintain the code. In NestJS, the DI system is built around the concept of providers and tokens.

Providers: A provider is a class, value, or factory function responsible for creating an object that can be injected into other classes. Providers are typically registered in the providers array of a module. Here's an example of a provider class:

    import { Injectable } from '@nestjs/common';

    @Injectable()
    export class CatsService {
      // Implementation of the service
    }

Tokens: Tokens are unique identifiers used by NestJS to locate providers. By default, a class’s token is the class itself. When a class needs a dependency, it requests it from the DI system using a token.
Injecting Dependencies: To inject a dependency, you can use the @Inject() decorator in the constructor of a class. The DI system will then resolve the dependency and provide an instance of the requested class. Here's an example:

    import { Injectable, Inject } from '@nestjs/common';
    import { CatsService } from './cats.service';

    @Injectable()
    export class AppService {
      constructor(private readonly catsService: CatsService) {}
    }

In this example, the AppService class requests an instance of CatsService in its constructor. The DI system locates the CatsService provider, creates an instance, and injects it into the AppService class.

Dependency Injection in NestJS provides the following benefits:

Decoupling: DI promotes loose coupling between classes by allowing them to depend on abstractions rather than concrete implementations. This makes it easier to modify or replace parts of the application without affecting other components.
Testing: DI makes it simpler to test classes by allowing you to replace dependencies with mock implementations during testing.
Reusability: By managing dependencies centrally, DI encourages the reuse of providers across multiple parts of the application.

Now that you understand Dependency Injection in NestJS, let’s discuss the role of pipes and their use cases. How can pipes help with validation and data transformation in a NestJS application?

to be continued

0–5 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 15:56:12 +0000

Modules and decorators are fundamental concepts in NestJS that contribute to its modular architecture, helping you create well-structured and maintainable applications.

Modules: In NestJS, a module is a class annotated with the @Module() decorator, responsible for organizing application components such as controllers, providers, and other modules. By grouping related components together, modules make it easier to manage large applications and promote code reusability.

A typical NestJS module looks like this:

    import { Module } from '@nestjs/common';
    import { CatsController } from './cats.controller';
    import { CatsService } from './cats.service';

    @Module({
      controllers: [CatsController],
      providers: [CatsService],
    })
    export class CatsModule {}

Decorators: Decorators are a TypeScript feature that allows you to add metadata to classes, properties, and methods. In NestJS, decorators are used extensively to enhance classes and provide additional information to the framework. Some common decorators in NestJS include @Controller(), @get(), @post(), and @Inject().

For instance, a basic NestJS controller using decorators could look like this:

    import { Controller, Get } from '@nestjs/common';

    @Controller('cats')
    export class CatsController {
      @Get()
      findAll() {
        return 'This action returns all cats.';
      }
    }

The @Controller('cats') decorator marks the CatsController class as a controller and associates it with the 'cats' route. The @get() decorator is applied to the findAll method, indicating that it should handle HTTP GET requests.

By leveraging modules and decorators, NestJS enforces a clean, modular structure for your application, making it easier to manage, extend, and maintain.

Now, let’s explore a crucial aspect of NestJS: dependency injection. Can you explain how dependency injection works in NestJS and how it helps with decoupling and testing?

to be continued

0–4 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 15:54:46 +0000

Interfaces in TypeScript are a powerful way to define contracts for the shape of objects, allowing you to enforce specific structures for objects and create reusable type definitions. They can also be used for type checking and autocompletion support in code editors.

Here’s an example of an interface in TypeScript:

    interface Person {
      firstName: string;
      lastName: string;
      age: number;
      greet(): string;
    }

This interface defines a Person object with firstName, lastName, and age properties, as well as a greet method. You can use the interface as a type for an object like this:

    const john: Person = {
      firstName: "John",
      lastName: "Doe",
      age: 30,
      greet() {
        return `Hello, my name is ${this.firstName} ${this.lastName}.`;
      },
    };

If you try to create an object that doesn’t match the Person interface, TypeScript will throw a type error during compilation.

Now let’s shift our focus to NestJS. Can you explain the importance of modules and decorators in NestJS and how they contribute to the framework’s modular architecture?

to be continued

0–3 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sat, 06 May 2023 15:56:26 +0000

Static typing and dynamic typing are two approaches to handling data types in programming languages.

In static typing, the data type of a variable is known at compile time, meaning that the type is explicitly specified or inferred when the code is written. This allows for type checking before the code is executed, which can help catch potential errors early and improve code reliability. Languages with static typing include TypeScript, Java, and C++.

In dynamic typing, the data type of a variable is determined at runtime, meaning that it’s determined when the code is executed. Variables can change types during the execution of a program, which can make the code more flexible but also more prone to errors. Languages with dynamic typing include JavaScript, Python, and Ruby.

Now that we’ve briefly discussed static typing, let’s delve into TypeScript basics. TypeScript introduces static typing to JavaScript, providing several data types such as number, string, boolean, null, and undefined. In addition, TypeScript has more advanced types like any, unknown, never, tuple, and enum.

For instance, declaring a variable with a specific type in TypeScript looks like this:

let age: number = 25;

If you try to assign a non-number value to the age variable, TypeScript will throw a type error during compilation.

Are you familiar with interfaces in TypeScript? If so, can you provide an example?

to be continued

0–2 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sat, 06 May 2023 15:55:24 +0000

Using TypeScript, NestJS, and Node.js together is beneficial for modern web development for several reasons:

Type Safety: TypeScript’s static typing helps catch potential errors during development, leading to more reliable and maintainable code. It can also enhance the development experience with better autocompletion and documentation support in code editors.
Consistent Language: Since Node.js allows you to write server-side code in JavaScript, using TypeScript across both the front-end and back-end of an application enables you to maintain a consistent language throughout the entire development stack.
Modular Architecture: NestJS’s modular approach, inspired by Angular, encourages a well-structured and organized codebase. This makes it easier to manage large applications and promotes code reusability.
Scalability and Performance: Node.js’s event-driven, non-blocking I/O model allows for efficient resource usage and high performance when handling multiple concurrent connections. NestJS enhances this by providing an abstraction layer that simplifies building scalable applications.
Growing Ecosystem: The combination of TypeScript, NestJS, and Node.js has a vibrant and growing ecosystem with a vast number of libraries, tools, and resources available to help you build powerful applications more quickly and efficiently.

Now that we’ve discussed the benefits of using TypeScript, NestJS, and Node.js together, let’s move on to exploring TypeScript basics. Are you familiar with static typing, and can you explain how it differs from dynamic typing?

to be continued

0–1 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sat, 06 May 2023 15:54:16 +0000

Let’s start with an introduction to TypeScript, NestJS, and Node.js! 😄

TypeScript is a superset of JavaScript that adds optional static typing to the language. This means that you can assign types to variables, function parameters, and return values, which helps you catch potential bugs early and improve the overall quality of your code. TypeScript compiles to plain JavaScript, making it compatible with any environment that supports JavaScript.

NestJS is a popular framework for building efficient, scalable Node.js server-side applications. It uses TypeScript by default and is built with a modular architecture, making it easy to organize your code and create reusable components. NestJS takes inspiration from Angular, bringing familiar concepts like decorators and dependency injection to the Node.js ecosystem.

Node.js is a runtime environment that allows you to run JavaScript on the server side. It’s built on the V8 JavaScript engine and uses an event-driven, non-blocking I/O model, which makes it lightweight and efficient for building scalable network applications.

Why do you think using TypeScript, NestJS, and Node.js together is beneficial for modern web development? 🤔

to be continued