TDD Your Way to Bulletproof Integrations

Roshan Raj — Tue, 27 May 2025 19:00:00 +0000

What We're Building

An event processing system that receives events, enriches them via external APIs, and publishes to an event bus. Ideal for demonstrating Test-Driven Development (TDD) in integration scenarios.

Event In → Transform → Call External API → Enrich → Publish to Event Bus

Why TDD for Integrations?

Integration code is notoriously difficult to test. External dependencies can fail, APIs can timeout, and data transformations can be complex. TDD forces us to think about these challenges upfront.

The TDD Journey

Step 1: Start Simple - Define the Contract

Red Phase: Write a test for the basic structure

test('should create processor with required dependencies', () => {
  const processor = new EventProcessor(mockApiClient, mockEventBus);
  expect(processor).toBeDefined();
});

Green Phase: Implement just enough to pass

class EventProcessor {
  constructor(apiClient, eventBus) {
    this.apiClient = apiClient;
    this.eventBus = eventBus;
  }
}

Key Insight: We've defined our dependencies through the test. The processor needs an API client and event bus.

Step 2: Test the Happy Path

Red Phase: Test basic event processing with mocks

test('should process and enrich user registration event', async () => {
  // Mock external dependencies
  const mockApiClient = {
    getUserProfile: jest.fn().mockResolvedValue({ tier: 'premium' })
  };

  // Test the transformation
  await processor.process(userRegistrationEvent);

  // Verify enrichment happened
  expect(mockEventBus.publish).toHaveBeenCalledWith(
    expect.objectContaining({ enrichedData: { tier: 'premium' } })
  );
});

Key Insight: Mocking reveals the interface we need from external systems.

Step 3: Handle Failures (The Real Value of TDD)

Red Phase: What happens when the API fails?

test('should handle API failure gracefully', async () => {
  mockApiClient.getUserProfile.mockRejectedValue(new Error('API timeout'));

  await expect(processor.process(event))
    .rejects.toThrow('Failed to process event');

  expect(mockEventBus.publish).not.toHaveBeenCalled();
});

Key Insight: TDD forces us to think about error scenarios before they happen in production.

Step 4: Add Retry Logic

Red Phase: Test that we retry on temporary failures

test('should retry API calls on failure', async () => {
  mockApiClient.getUserProfile
    .mockRejectedValueOnce(new Error('Temporary failure'))
    .mockResolvedValueOnce({ tier: 'basic' });

  await processor.process(event);

  expect(mockApiClient.getUserProfile).toHaveBeenCalledTimes(2);
  expect(mockEventBus.publish).toHaveBeenCalled();
});

Key Insight: The test drives us to implement retry logic with configurable attempts.

Step 5: Validate Input

Red Phase: Test validation before processing

test('should validate required event fields', async () => {
  const invalidEvent = { eventType: 'USER_REGISTERED' }; // missing userId

  await expect(processor.process(invalidEvent))
    .rejects.toThrow('Invalid event: missing userId');
});

Key Insight: TDD helps identify validation needs early.

Step 6: Handle Different Event Types

Red Phase: Test event-specific transformations

test('should apply different transformations based on event type', async () => {
  // For ORDER_PLACED events, we need both user and order data
  mockApiClient.getOrderDetails.mockResolvedValue({ items: [...] });

  await processor.process(orderEvent);

  expect(mockApiClient.getUserProfile).toHaveBeenCalled();
  expect(mockApiClient.getOrderDetails).toHaveBeenCalled();
});

Key Insight: Tests reveal that different events need different enrichment strategies.

Step 7: Add Dead Letter Queue

Red Phase: Test handling of persistent failures

test('should send to DLQ after all retries fail', async () => {
  mockApiClient.getUserProfile
    .mockRejectedValue(new Error('Service unavailable'));

  await processor.process(event);

  expect(mockDLQ.send).toHaveBeenCalledWith({
    originalEvent: event,
    error: expect.stringContaining('after 3 attempts')
  });
});

Key Insight: TDD drives us to implement proper failure handling strategies.

The Final Architecture

Through TDD, we've built a robust event processor with:

✅ Input validation
✅ Retry logic with configurable attempts
✅ Different transformation strategies per event type
✅ Dead letter queue for failed events
✅ Proper error handling and logging

Key Lessons from TDD Integration Development

1. Start with the Contract

Tests define what your external dependencies should provide. This makes integration points explicit.

2. Embrace Mocking

Mocks aren't just for isolation - they're design tools that help you discover the right interfaces.

3. Error Paths First

TDD forces you to think about failures before successes. In integration code, this is invaluable.

4. Progressive Enhancement

Each test adds a new capability. You build complexity incrementally, not all at once.

5. Tests as Documentation

Your test suite becomes living documentation of how the system handles various scenarios.

TDD Benefits for Integration Code

Confidence in Error Handling: You've tested failures before they happen
Clear Dependencies: Tests make external dependencies explicit
Refactoring Safety: Comprehensive tests let you improve code fearlessly
Better Design: TDD drives you toward loosely coupled, testable code
Reduced Debugging: Issues are caught during development, not in production

When to Write Integration Tests vs Unit Tests

Unit Tests (with mocks):

Business logic validation
Transformation rules
Error handling flows
Retry mechanisms

Integration Tests:

End-to-end flow validation
Contract verification
Performance characteristics
Real failure scenarios

Common Pitfalls to Avoid

Over-mocking: Don't mock internal implementation details
Testing Implementation: Focus on behavior, not how it's coded
Skipping Error Cases: These are often more important than happy paths
Tight Coupling to Tests: Keep tests focused on public interfaces

Conclusion

TDD transforms integration development from a debugging nightmare into a systematic process. By writing tests first, we:

Design better interfaces
Handle errors gracefully
Build confidence in our code
Create maintainable systems

The key is to start simple, test one behavior at a time, and let the tests guide your design. Your future self (and your ops team) will thank you when those 3 AM integration failures don't happen because you've already tested for them.

Deploy React SPA to CloudFront and S3

Roshan Raj — Sat, 06 Aug 2022 09:10:10 +0000

An SPA (Single-page application) is a web app implementation that loads only a single web document, and then updates the body content of that single document via JavaScript APIs such as XMLHttpRequest and Fetch when different content is to be shown. Read more about it here.

For the purpose of this post, I will be using a simple react app.

Setting up S3 bucket to hold our react build.

Create a bucket

Keep the bucket name same as the domain (Helps).
Let the Object Ownership be 'ACLs Disabled'
Block All Public Access.

Deploy the React Application Build Folder to the S3 bucket

Make sure the index.html file is in the root of the directory.

Setting up CloudFront

Create a new CloudFront Distribution

Select the Origin as the S3 bucket we just created.
S3 Bucket Access: If you do not already have an Object Access Identity, create a new one.
We can let CloudFront update the policy for us. (Select Yes, update the bucket policy)
Price Class: You can select the regions as per your needs.
Since we will be using this CloudFront in our Domain Name, Provide the CNAME Domain Name. My Domain name is roshan-raj.com.
You can choose to add a custom SSL certificate. You can choose to request one from AWS. This will be free of cost if used in CloudFront. If you are creating a new SSL Certificate, it needs to be in the N.Virginia Region.
Specify the Default Root Object as index.html.
You can turn on Standard Logging. (Recommended)

Going Back to S3.

CloudFront should have added a policy for our bucket.
We need to add another statement in the Policy, which is s3:ListBucket.

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <CLOUDFRONT>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::roshan-raj/*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <CLOUDFRONT>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::roshan-raj"
        }
    ]
}

We are done. The React App will be available at the CloudFront URL.

Now, If you have any further routes in the React SPA, going to that routes will throw an Error "NoSuchKey".

We can fix this using a CloudFront Function, to re-write the url and attaching an index.html to the URL.

Create a CloudFront function and use the below code. https://github.com/aws-samples/amazon-cloudfront-functions/blob/main/url-rewrite-single-page-apps/index.js
Publish the function and attach it to the CloudFront > Behaviour > Edit > Viewer Request.

The "NoSuchKey" error is now fixed.
To connect it to a domain on Route53, simple point your domain to the CloudFront Domain Name.

Forem: Roshan Raj