Forem: Sheikh Shahzaman

How Architecture Tests Caught a Data Isolation Bug Before It Reached Production

Sheikh Shahzaman — Sat, 25 Apr 2026 09:52:39 +0000

TL;DR: Pest PHP architecture tests let you write your team coding rules as automated tests. Controllers using the DB facade directly? Build fails. Model missing the tenant isolation trait? Build fails. These tests caught a critical data isolation bug three weeks after we added them.

We had a rule in our team wiki. Every model that stores tenant-specific data must use the BelongsToTenant trait. This trait adds a global scope that automatically filters all queries by the current tenant. Without it a query could accidentally return data from other tenants.

The rule was clear. Everyone knew about it. It was mentioned in onboarding. It was in our code review checklist.

A new developer joined the team. Three weeks later they added a new model without the trait. The code review missed it because the reviewer was focused on the business logic which was actually well written. The model went to production.

For two days one tenant could see fragments of another tenant data in a specific report. We caught it from a support ticket not from our tests.

That was the day I added architecture tests.

What Architecture Tests Look Like

Pest PHP includes an arch() function that lets you write assertions about your code structure. Not about what your code does but about how it is organized.

// tests/Architecture/ArchTest.php

arch('models must use BelongsToTenant trait')
    ->expect('App\Models')
    ->toUseTrait('App\Traits\BelongsToTenant');

arch('controllers should not use DB facade')
    ->expect('App\Http\Controllers')
    ->not->toUse('Illuminate\Support\Facades\DB');

arch('services should not use Request')
    ->expect('App\Services')
    ->not->toUse('Illuminate\Http\Request');

arch('value objects must be readonly')
    ->expect('App\ValueObjects')
    ->toBeReadonly();

arch('soft delete models must also have tenant scope')
    ->expect('App\Models')
    ->toUseTrait('Illuminate\Database\Eloquent\SoftDeletes')
    ->toUseTrait('App\Traits\BelongsToTenant');

These tests run in our CI pipeline on every commit. If any rule is violated the build fails. The developer gets a clear error message telling them exactly which rule they broke and which file broke it.

The Tests That Have Saved Us

Here are the architecture tests that have caught the most violations in the last 6 months.

The tenant isolation test caught 4 new models that were missing the BelongsToTenant trait. Each one would have been a potential data leak. This single test justified the entire effort.

The controller purity test catches developers who write database queries directly in controllers instead of using service classes. This happens more often than you would think especially when someone is "just adding a quick endpoint."

// This would fail the architecture test
class ReportController extends Controller {
    public function index() {
        $data = DB::table('orders')
            ->where('tenant_id', auth()->user()->tenant_id)
            ->sum('total');

        return response()->json($data);
    }
}

The right approach is moving that query into a service class. The controller just calls the service and returns the result. This keeps controllers thin and makes the business logic testable in isolation.

The dependency direction test ensures that inner layers never depend on outer layers. Services do not import Request objects. Models do not import controller classes. This keeps the dependency graph clean and makes the code easier to test.

Going Beyond Basic Rules

The basic trait and dependency checks are useful but the real power comes from combining architecture tests with custom logic.

arch('controller methods should use Form Requests')
    ->expect('App\Http\Controllers')
    ->not->toUse('Illuminate\Http\Request');

arch('no env calls outside config')
    ->expect('App')
    ->not->toUse(['env']);

The env() test is one I wish I had added earlier. Calling env() directly in application code works in development but returns null when the config is cached in production. This is a classic Laravel gotcha that catches developers at least once. The architecture test prevents it permanently.

I also added a test that ensures no model uses the $guarded property set to an empty array. We had an incident where a mass assignment vulnerability slipped through because a developer set $guarded to an empty array thinking it was the same as setting $fillable. It is not. An empty $guarded means every field is mass assignable.

arch('models should not use empty guarded')
    ->expect('App\Models')
    ->not->toUseStrictlyEmptyGuarded();

A Mistake I Made Along the Way

When I first set up architecture tests I went overboard. I added 20 rules in one pull request. The CI pipeline lit up like a Christmas tree. Over 150 violations across the codebase.

The team was annoyed. Nobody wanted to fix 150 violations that had nothing to do with their current feature work. I had to revert the PR and take a different approach.

The lesson was clear. You can not enforce architectural rules retroactively on a large codebase all at once. It does not work socially even if it works technically.

How to Introduce Architecture Tests to an Existing Project

Here is the approach that worked for us after I learned from that mistake.

Step 1. Start with the most critical rule only. For us that was the tenant isolation trait. One test. Fix the handful of violations. Merge it.

Step 2. Add one new rule per week. Give the team time to adjust and fix existing violations.

Step 3. For rules with many existing violations use the ignoring() method to exclude legacy code.

arch('controllers should not use DB')
    ->expect('App\Http\Controllers')
    ->not->toUse('Illuminate\Support\Facades\DB')
    ->ignoring('App\Http\Controllers\Legacy');

This lets you enforce the rule for all new code while giving the team time to refactor old code.

Step 4. Track violations over time. We added a simple script that counts the number of ignoring() exceptions. That number should go down every sprint not up. We went from 47 exceptions down to 12 over four months. The trend line matters more than the absolute number.

Real Numbers from Production

After running architecture tests for 6 months the results spoke for themselves.

The tenant isolation test caught 4 models missing the BelongsToTenant trait. Each would have been a data leak between tenants. Response time to catch these dropped from "whenever someone notices in production" to "immediately in CI."

The controller purity test caught 23 violations in the first month alone. After 3 months that number dropped to 2 per month. Developers learned the pattern and stopped writing database queries in controllers.

The env() test caught 7 direct calls that would have broken in production when config caching was enabled. One of those was in the payment processing flow. That single catch probably saved us a weekend of debugging.

What I Would Do Differently

I would add architecture tests on day one of every new project. Adding them to an existing codebase is painful because of all the existing violations. Starting fresh means the rules are enforced from the first commit and there is never any catch-up work.

I also would have been more aggressive about testing dependency directions. We caught the DB-in-controller pattern early but we were slow to add tests for other dependency violations. By the time we added them there were 40+ violations to fix.

And I would create a shared package of common architecture tests that every project uses. We ended up writing very similar tests across three different projects. A reusable test suite would have saved time. I have not built that package yet but it is on my list.

The Bigger Picture

Architecture tests are not about being strict for the sake of being strict. They are about making your codebase predictable.

When every model has the tenant trait you know data isolation is enforced everywhere. When no controller touches the database directly you know the service layer is the single source of truth. When dependencies flow in one direction you know refactoring one layer will not break another.

These guarantees compound over time. Six months in the codebase is still clean because the tests prevent it from degrading. New developers write code that follows the patterns automatically because the CI pipeline will not let them do otherwise.

Your team rules should live in your test suite not in a wiki.

What architecture rules would you enforce if you had automated tests for them?

Why We Switched from Direct API Calls to Kafka and What Broke Along the Way

Sheikh Shahzaman — Sat, 25 Apr 2026 09:38:22 +0000

TL;DR: We migrated 10+ microservices from direct HTTP calls to Kafka event-driven communication. Reliability improved massively but the migration was harder than expected. Here are the real lessons including the mistakes.

Our system started as a monolith. Then we split it into microservices. The services talked to each other using direct HTTP calls. Service A would POST to Service B which would POST to Service C. It worked fine when we had 3 services.

Then we had 10.

The Day Everything Cascaded

One Tuesday morning our notification service crashed because of a memory leak. No big deal right? Restart it and move on.

But the order service was calling the notification service directly during checkout. When notification service was down the order endpoint started timing out. Users could not place orders. The billing service was also calling notification service to confirm payment receipts. Billing started failing too.

One crashed service took down three other services because they were all directly dependent on it.

That was the day we decided to move to event-driven architecture.

How We Set It Up

The concept is simple. Instead of Service A calling Service B directly Service A publishes an event to Kafka. Service B listens for that event and processes it on its own time.

// Before: Direct coupling
class OrderService {
    public function complete($order) {
        $order->markComplete();
        Http::post('billing-service/invoice', $order->toArray());
        Http::post('notification-service/email', $order->toArray());
        Http::post('analytics-service/track', $order->toArray());
    }
}

// After: Event-driven
class OrderService {
    public function complete($order) {
        $order->markComplete();
        KafkaProducer::publish('order.completed', [
            'order_id' => $order->id,
            'tenant_id' => $order->tenant_id,
            'total' => $order->total,
            'completed_at' => now()->toIso8601String(),
        ]);
    }
}

The order service does not know or care who listens to that event. Billing creates an invoice. Notifications send an email. Analytics tracks a metric. Each service subscribes to the event independently.

What Broke During Migration

I wish I could say the migration was smooth. It was not.

Problem 1: Event ordering. We assumed events would arrive in the order they were published. They mostly did. But when we had high throughput some consumers processed events out of order. An "order.updated" event arrived before "order.created" and the consumer crashed because the order did not exist yet.

The fix was adding an event version number and having consumers check if they had already processed a newer version before applying changes.

Problem 2: Duplicate events. Kafka guarantees at-least-once delivery. That means consumers can receive the same event twice. We had a bug where a payment was processed twice because the consumer was not idempotent.

The fix was adding a unique event ID and checking if we had already processed that ID before taking action.

class InvoiceConsumer {
    public function handle($event) {
        if (ProcessedEvent::where('event_id', $event['id'])->exists()) {
            return;
        }

        $invoice = Invoice::createFromOrder($event['order_id']);

        ProcessedEvent::create(['event_id' => $event['id']]);
    }
}

Problem 3: Debugging was harder. With direct API calls you could trace a request from start to finish in one log. With events the flow is split across multiple services and multiple time periods. Finding out why an invoice was not created required checking logs in three different services.

We solved this by adding a correlation ID to every event. When the order service publishes an event it includes a unique request ID. Every downstream consumer includes that same ID in their logs. Now you can search for one ID and see the entire flow across all services.

The Patterns That Saved Us

Dead letter queue. When a consumer fails to process an event after 3 retries it goes to a dead letter topic. We have a dashboard that shows failed events and lets us replay them after fixing the bug.

Schema registry. We define the structure of every event in a shared schema. If a producer tries to publish an event that does not match the schema it fails at publish time not at consume time. This prevented so many bugs.

Consumer lag monitoring. We track how far behind each consumer is. If the notification consumer falls 10,000 events behind we get an alert. This caught performance issues before users noticed them.

The Results

After 3 months on event-driven architecture:

Zero cascading failures. One service going down does not affect any other service.
We can deploy services independently without coordinating with other teams.
Adding a new consumer takes 30 minutes instead of modifying 5 different services.
Event replay lets us reprocess historical data when we add new features.

What I Would Do Differently

I would have implemented idempotency from day one not after the duplicate payment bug. Every consumer should be idempotent by default.

I would have invested in better tooling earlier. A good event viewer that shows the flow of events across services would have saved weeks of debugging time.

And I would not have migrated everything at once. We tried to move all 10 services in one sprint. It should have been gradual. Start with the least critical services and work toward the most critical.

Event-driven architecture is powerful but it adds complexity. If you have 3 services that rarely fail direct API calls are probably fine. If you have 10+ services and reliability matters events are worth the investment.

Have you migrated from direct API calls to event-driven architecture? What surprised you the most?

Why We Switched from Direct API Calls to Kafka and What Broke Along the Way

Sheikh Shahzaman — Tue, 21 Apr 2026 05:21:38 +0000

Then we had 10.

The Day Everything Cascaded

One Tuesday morning our notification service crashed because of a memory leak. No big deal right? Restart it and move on.

One crashed service took down three other services because they were all directly dependent on it.

That was the day we decided to move to event-driven architecture.

How We Set It Up

The concept is simple. Instead of Service A calling Service B directly Service A publishes an event to Kafka. Service B listens for that event and processes it on its own time.

// Before: Direct coupling
class OrderService {
    public function complete($order) {
        $order->markComplete();
        Http::post('billing-service/invoice', $order->toArray());
        Http::post('notification-service/email', $order->toArray());
        Http::post('analytics-service/track', $order->toArray());
    }
}

// After: Event-driven
class OrderService {
    public function complete($order) {
        $order->markComplete();
        KafkaProducer::publish('order.completed', [
            'order_id' => $order->id,
            'tenant_id' => $order->tenant_id,
            'total' => $order->total,
            'completed_at' => now()->toIso8601String(),
        ]);
    }
}

What Broke During Migration

I wish I could say the migration was smooth. It was not.

The fix was adding an event version number and having consumers check if they had already processed a newer version before applying changes.

The fix was adding a unique event ID and checking if we had already processed that ID before taking action.

class InvoiceConsumer {
    public function handle($event) {
        if (ProcessedEvent::where('event_id', $event['id'])->exists()) {
            return;
        }

        $invoice = Invoice::createFromOrder($event['order_id']);

        ProcessedEvent::create(['event_id' => $event['id']]);
    }
}

The Patterns That Saved Us

The Results

After 3 months on event-driven architecture:

Zero cascading failures. One service going down does not affect any other service.
We can deploy services independently without coordinating with other teams.
Adding a new consumer takes 30 minutes instead of modifying 5 different services.
Event replay lets us reprocess historical data when we add new features.

What I Would Do Differently

I would have implemented idempotency from day one not after the duplicate payment bug. Every consumer should be idempotent by default.

I would have invested in better tooling earlier. A good event viewer that shows the flow of events across services would have saved weeks of debugging time.

Have you migrated from direct API calls to event-driven architecture? What surprised you the most?

Why We Switched from Direct API Calls to Kafka and What Broke Along the Way

Sheikh Shahzaman — Wed, 15 Apr 2026 01:45:38 +0000

Then we had 10.

The Day Everything Cascaded

One Tuesday morning our notification service crashed because of a memory leak. No big deal right? Restart it and move on.

One crashed service took down three other services because they were all directly dependent on it.

That was the day we decided to move to event-driven architecture.

How We Set It Up

The concept is simple. Instead of Service A calling Service B directly Service A publishes an event to Kafka. Service B listens for that event and processes it on its own time.

// Before: Direct coupling
class OrderService {
    public function complete($order) {
        $order->markComplete();
        Http::post('billing-service/invoice', $order->toArray());
        Http::post('notification-service/email', $order->toArray());
        Http::post('analytics-service/track', $order->toArray());
    }
}

// After: Event-driven
class OrderService {
    public function complete($order) {
        $order->markComplete();
        KafkaProducer::publish('order.completed', [
            'order_id' => $order->id,
            'tenant_id' => $order->tenant_id,
            'total' => $order->total,
            'completed_at' => now()->toIso8601String(),
        ]);
    }
}

What Broke During Migration

I wish I could say the migration was smooth. It was not.

The fix was adding an event version number and having consumers check if they had already processed a newer version before applying changes.

The fix was adding a unique event ID and checking if we had already processed that ID before taking action.

class InvoiceConsumer {
    public function handle($event) {
        if (ProcessedEvent::where('event_id', $event['id'])->exists()) {
            return;
        }

        $invoice = Invoice::createFromOrder($event['order_id']);

        ProcessedEvent::create(['event_id' => $event['id']]);
    }
}

The Patterns That Saved Us

The Results

After 3 months on event-driven architecture:

Zero cascading failures. One service going down does not affect any other service.
We can deploy services independently without coordinating with other teams.
Adding a new consumer takes 30 minutes instead of modifying 5 different services.
Event replay lets us reprocess historical data when we add new features.

What I Would Do Differently

I would have implemented idempotency from day one not after the duplicate payment bug. Every consumer should be idempotent by default.

I would have invested in better tooling earlier. A good event viewer that shows the flow of events across services would have saved weeks of debugging time.

Have you migrated from direct API calls to event-driven architecture? What surprised you the most?

How defer() in Laravel 11 Made Our API Responses 3x Faster Without Touching a Single Queue

Sheikh Shahzaman — Sun, 29 Mar 2026 12:17:24 +0000

TL;DR: Laravel 11 introduced defer() which runs code after the HTTP response is sent to the user. No queues, no job classes, no workers. Just wrap your fire-and-forget logic in defer() and your API becomes instantly faster.

I spent two days last year trying to figure out why our order API endpoint was taking 1.2 seconds to respond. The order itself was being created in about 80ms. So where was the rest of the time going?

Turns out we were sending a confirmation email, tracking an analytics event, syncing inventory with a third-party service, and clearing a cache key. All of that happened synchronously before the response was sent back to the user.

The user did not care about any of those things completing before they saw their order confirmation. They just wanted to know their order went through.

The Old Way: Queues for Everything

The typical advice is to push these tasks onto a queue. Create a job class, dispatch it, run a queue worker, set up monitoring, handle failed jobs. For a large application with complex background processing that makes sense.

But for simple fire-and-forget tasks like sending an email or tracking an event? That is a lot of infrastructure for something that should be simple.

We had 14 different job classes in our application. Eight of them were single-method classes that just did one small thing. Each one had its own file, its own test, and its own entry in the failed jobs table. It felt like overkill.

Enter defer()

Laravel 11 added defer() and it changed how I think about background tasks. Here is how it works:

Route::post('/order', function () {
    $order = Order::create($data);

    defer(fn() => Mail::send(new OrderConfirmation($order)));
    defer(fn() => Analytics::track('order_placed', $order));
    defer(fn() => InventorySync::push($order));
    defer(fn() => Cache::forget("user:{$order->user_id}:cart"));

    return response()->json($order);
});

The response goes back to the user immediately after the order is created. Then Laravel runs all the deferred callbacks after the response is sent. The user never waits for them.

No job classes. No queue workers. No Redis or database queue driver. Just a closure that runs after the response.

When To Use defer() vs Queues

This is the part most articles get wrong. They either say "use defer for everything" or "always use queues." The reality is more nuanced.

Use defer() when the task is simple and does not need retry logic. Sending a notification email, tracking an analytics event, clearing a cache, logging an activity. If it fails you do not need to retry it automatically.

Use queues when the task is complex or needs reliability guarantees. Processing a payment, generating a large PDF, syncing thousands of records with an external API. If it fails you need to know about it and retry it.

I made the mistake of using defer() for a webhook delivery early on. The webhook target was unreliable and about 10% of deliveries failed silently. There was no retry mechanism, no failed job record, no way to know it happened. I moved that back to a queue with 3 retries and the problem was solved.

Real Numbers

After refactoring our order endpoint to use defer() for non-critical tasks:

Response time dropped from 1.2 seconds to 280ms
User-perceived performance improved dramatically
We deleted 8 single-method job classes
Queue worker load decreased because fewer jobs were being dispatched

The email still sends. The analytics event still tracks. The cache still clears. The user just does not wait for any of it.

A Pattern I Use in Every Project

I created a simple middleware that defers common request-level tasks:

class DeferCommonTasks
{
    public function handle($request, Closure $next)
    {
        $response = $next($request);

        defer(fn() => ActivityLog::record($request));
        defer(fn() => MetricsCollector::trackRequest($request, $response));

        return $response;
    }
}

Every request automatically logs activity and tracks metrics without adding any latency to the response. The middleware runs once and every endpoint benefits from it.

What I Would Do Differently

If I started over I would establish a clear rule from day one: if the task does not need retry logic and takes less than 5 seconds, use defer(). If it needs retries or takes longer, use a queue. Having that rule early would have prevented the 14 unnecessary job classes we ended up with.

I also would not use defer() inside database transactions. If the transaction rolls back the deferred callback still runs because it executes after the response. This caused a bug where we were sending order confirmation emails for orders that failed to save. I learned that the hard way.

The Bottom Line

defer() is not a replacement for queues. It is a replacement for the dozens of tiny job classes that exist only because you needed something to run "after" the response.

One line of code. No infrastructure changes. Measurably faster responses.

If you are on Laravel 11 or later and you are not using defer() yet you are probably making your users wait for things they do not need to wait for.

What is the simplest performance win you have found in your Laravel applications?

How defer() in Laravel 11 Made Our API Responses 3x Faster Without Touching a Single Queue

Sheikh Shahzaman — Sun, 29 Mar 2026 01:19:09 +0000

TL;DR: Laravel 11 introduced defer() which runs code after the HTTP response is sent to the user. No queues, no job classes, no workers. Just wrap your fire-and-forget logic in defer() and your API becomes instantly faster.

The user did not care about any of those things completing before they saw their order confirmation. They just wanted to know their order went through.

The Old Way: Queues for Everything

But for simple fire-and-forget tasks like sending an email or tracking an event? That is a lot of infrastructure for something that should be simple.

Enter defer()

Laravel 11 added defer() and it changed how I think about background tasks. Here is how it works:

Route::post('/order', function () {
    $order = Order::create($data);

    defer(fn() => Mail::send(new OrderConfirmation($order)));
    defer(fn() => Analytics::track('order_placed', $order));
    defer(fn() => InventorySync::push($order));
    defer(fn() => Cache::forget("user:{$order->user_id}:cart"));

    return response()->json($order);
});

The response goes back to the user immediately after the order is created. Then Laravel runs all the deferred callbacks after the response is sent. The user never waits for them.

No job classes. No queue workers. No Redis or database queue driver. Just a closure that runs after the response.

When To Use defer() vs Queues

This is the part most articles get wrong. They either say "use defer for everything" or "always use queues." The reality is more nuanced.

Real Numbers

After refactoring our order endpoint to use defer() for non-critical tasks:

Response time dropped from 1.2 seconds to 280ms
User-perceived performance improved dramatically
We deleted 8 single-method job classes
Queue worker load decreased because fewer jobs were being dispatched

The email still sends. The analytics event still tracks. The cache still clears. The user just does not wait for any of it.

A Pattern I Use in Every Project

I created a simple middleware that defers common request-level tasks:

class DeferCommonTasks
{
    public function handle($request, Closure $next)
    {
        $response = $next($request);

        defer(fn() => ActivityLog::record($request));
        defer(fn() => MetricsCollector::trackRequest($request, $response));

        return $response;
    }
}

Every request automatically logs activity and tracks metrics without adding any latency to the response. The middleware runs once and every endpoint benefits from it.

What I Would Do Differently

The Bottom Line

defer() is not a replacement for queues. It is a replacement for the dozens of tiny job classes that exist only because you needed something to run "after" the response.

One line of code. No infrastructure changes. Measurably faster responses.

If you are on Laravel 11 or later and you are not using defer() yet you are probably making your users wait for things they do not need to wait for.

What is the simplest performance win you have found in your Laravel applications?