Forem: Juan Carlos González Cabrero

Decoupling the Core: An Introduction to Hexagonal Architecture

Juan Carlos González Cabrero — Fri, 20 Mar 2026 12:58:56 +0000

When Layered Architecture Stops Scaling

Layered architecture usually looks fine in the first month. The trouble starts when the service stops being a CRUD API and begins coordinating external providers, scheduled jobs, cache layers, persistence rules, and business constraints that are no longer trivial. At that point, the classic controller-service-repository stack tends to accumulate responsibilities in the wrong places.

That is where Hexagonal Architecture starts to make sense. Not because the diagram is elegant, but because it gives the design a rule that is easy to reason about: business logic stays in the center, infrastructure stays at the edges, and dependencies point inward.

Why classic layering breaks down

Traditional layered systems are not wrong. They are just easier to get wrong as the application grows.

The usual failure mode is predictable:

controllers start orchestrating use cases
services become transaction scripts with framework knowledge
repositories stop being persistence abstractions and begin to carry business decisions
domain objects degrade into DTOs

The real problem is not the number of layers. The real problem is dependency direction. Once domain logic depends on JPA entities, Spring annotations, HTTP DTOs, or provider payloads, changing any outer concern becomes more expensive than it should be.

That cost shows up in very practical ways:

tests become slow because business rules need full application context
switching storage technology becomes invasive
adding a second delivery mechanism, such as messaging or scheduling, duplicates orchestration logic
external integration concerns leak into the core model

Hexagonal Architecture is a response to that coupling problem.

The central idea of Hexagonal Architecture

The pattern is simpler than the terminology sometimes suggests.

A hexagonal application has:

a core with domain and use cases
inbound ports, which define what the application can do
outbound ports, which define what the application needs
adapters, which connect those ports to HTTP, databases, queues, schedulers, external APIs, and any other technical detail

The point is not to create more interfaces for the sake of it. The point is to isolate the places where change is likely and costly.

Here is the idea in one sentence: the application core should not know how data arrives, how it is stored, or which library is doing the work outside the boundary.

Inbound and outbound ports

Inbound ports model use cases:

public interface SearchSlotsUseCase {
    List<PerformanceSlot> searchPublishedSlots(OffsetDateTime opensAfter, OffsetDateTime closesBefore);
}

public interface RefreshSlotCatalogUseCase {
    void refreshCatalog();
}

Outbound ports model dependencies:

public interface SlotQueryRepository {
    List<PerformanceSlot> findPublishedSlots(SlotSearchCriteria criteria);
}

public interface SlotCommandRepository {
    void mergeSnapshot(List<PerformanceSlot> slots, Instant importedAt);
    void disableMissingSlots(Instant importedAt);
}

public interface PartnerCatalogClient {
    List<PerformanceSlot> fetchSlots();
}

Those interfaces are already saying something important. The application depends on business capabilities, not on JpaRepository, RestTemplate, Redis commands, or XML parsers.

Hexagonal Architecture and DDD fit naturally together

Hexagonal Architecture and Domain-Driven Design solve different problems, but they work well together.

Hexagonal Architecture answers: where should dependencies point?

DDD answers: how should the domain be modeled?

In practice, the combination is useful because a clean architecture without a meaningful domain model quickly becomes ceremony, and a rich domain model without clear boundaries tends to get contaminated by infrastructure concerns.

What DDD looks like in a service like this

You do not need the full DDD catalogue to get value. A few tactical choices are enough:

name domain concepts in business language
model small value objects when identity or meaning matters
keep invariants close to the model
represent domain errors explicitly

For example, if a slot coming from a partner catalog is identified by two related ids, that pair deserves its own type:

public record PartnerSlotId(String seasonCode, Long slotNumber) {

    public PartnerSlotId {
        if (seasonCode == null || seasonCode.isBlank() || slotNumber == null) {
            throw new DomainValidationException("Partner slot id is required");
        }
    }
}

That is a better model than passing two unrelated Long values through every layer and hoping nobody swaps their order.

The same applies to the aggregate root or main entity:

public record PerformanceSlot(
        UUID id,
        PartnerSlotId partnerSlotId,
        String headline,
        LocalDateTime opensAt,
        LocalDateTime closesAt,
        LocalDateTime bookingStartsAt,
        LocalDateTime bookingEndsAt,
        SalesChannel salesChannel,
        boolean soldOut,
        List<SeatBand> seatBands
) {

    public PerformanceSlot {
        if (headline == null || headline.isBlank()) {
            throw new DomainValidationException("headline is required");
        }
        if (opensAt == null || closesAt == null) {
            throw new DomainValidationException("slot dates are required");
        }
        if (opensAt.isAfter(closesAt)) {
            throw new InvalidDateRangeException("opensAt cannot be after closesAt");
        }
        if (bookingStartsAt != null && bookingEndsAt != null && bookingStartsAt.isAfter(bookingEndsAt)) {
            throw new InvalidDateRangeException("bookingStartsAt cannot be after bookingEndsAt");
        }
        seatBands = seatBands == null ? List.of() : List.copyOf(seatBands);
    }
}

This is where DDD pulls its weight. The model is not just data storage. It protects meaning and rejects invalid states.

A pragmatic warning about DDD

This is also where many teams overdo it.

Not every object needs to become a rich domain object. Not every service needs domain events, factories, specifications, and anti-corruption layers from day one. If the domain is modest, forcing every DDD pattern into the design usually makes the code harder to follow.

The useful test is simple: does this abstraction make the business model clearer and safer, or is it only there because the pattern exists?

A Spring Boot structure that stays readable

A practical structure for a hexagonal Spring Boot service usually looks like this:

src/main/java/com/example/catalog
├── domain
│   ├── exception
│   ├── model
│   └── port
├── application
│   ├── port/in
│   └── service
└── infrastructure
    ├── config
    ├── inbound
    └── outbound

The package names matter less than the rule behind them. Domain and application should not depend on infrastructure. Infrastructure is allowed to depend on the core.

I also think it is worth being slightly strict here: if your domain package imports JPA annotations or Spring MVC types, the architecture is already drifting.

The application layer should orchestrate, not absorb infrastructure

The read side can stay very small:

@Service
@RequiredArgsConstructor
public class SearchSlotsService implements SearchSlotsUseCase {

    private final SlotQueryRepository slotQueryRepository;

    @Override
    @Transactional(readOnly = true)
    public List<PerformanceSlot> searchPublishedSlots(OffsetDateTime opensAfter, OffsetDateTime closesBefore) {
        SlotSearchCriteria criteria = SlotSearchCriteria.of(
                opensAfter == null ? null : opensAfter.toLocalDateTime(),
                closesBefore == null ? null : closesBefore.toLocalDateTime()
        );

        return slotQueryRepository.findPublishedSlots(criteria);
    }
}

This is the right level of responsibility for an application service. It coordinates the use case, translates raw input into a domain concept, and delegates the actual query to an outbound port.

The write side usually shows the architecture more clearly:

@Service
@RequiredArgsConstructor
public class RefreshSlotCatalogService implements RefreshSlotCatalogUseCase {

    private final PartnerCatalogClient partnerCatalogClient;
    private final SlotCommandRepository slotCommandRepository;
    private final Clock clock;

    @Override
    public void refreshCatalog() {
        Instant importedAt = Instant.now(clock);
        List<PerformanceSlot> slots = partnerCatalogClient.fetchSlots();

        slotCommandRepository.mergeSnapshot(slots, importedAt);
        slotCommandRepository.disableMissingSlots(importedAt);
    }
}

There are three details here that are easy to miss and worth keeping:

First, Clock is injected directly. I prefer that over inventing a ClockPort unless time itself is a business capability. Not every technical dependency deserves its own architectural boundary.

Second, the write port exposes business operations, not generic CRUD verbs. mergeSnapshot and disableMissingSlots describe a synchronization process much better than save() ever could.

Third, the service has no idea whether the provider uses XML, JSON, HTTP, gRPC, or a message queue. That uncertainty stays outside.

Adapters should translate, not think

This is one of the most useful practical rules in a hexagonal system.

Adapters are translators. They convert transport models into domain objects and domain objects into transport or persistence models. They should be as boring as possible.

A persistence adapter can map between the domain model and JPA entities:

@Component
@RequiredArgsConstructor
public class JpaSlotCommandRepositoryAdapter implements SlotCommandRepository {

    private final SpringDataSlotRepository repository;
    private final SlotEntityMapper mapper;

    @Override
    public void mergeSnapshot(List<PerformanceSlot> slots, Instant importedAt) {
        List<SlotEntity> entities = slots.stream()
                .map(slot -> mapper.toEntity(slot, importedAt))
                .toList();

        repository.saveAll(entities);
    }

    @Override
    public void disableMissingSlots(Instant importedAt) {
        repository.disableMissingSlots(importedAt);
    }
}

A provider adapter can isolate HTTP and parsing concerns:

@Component
@RequiredArgsConstructor
public class PartnerCatalogClientAdapter implements PartnerCatalogClient {

    private final PartnerFeedGateway partnerFeedGateway;
    private final PartnerSlotFeedParser partnerSlotFeedParser;

    @Override
    public List<PerformanceSlot> fetchSlots() {
        String payload = partnerFeedGateway.fetchPayload();
        return partnerSlotFeedParser.parse(payload);
    }
}

This split matters because retry policy, HTTP timeouts, payload validation, caching, and fetch strategy are not domain concerns. They change for different reasons and should live in different adapters or configuration classes.

CQRS often appears naturally here

A lot of articles treat CQRS like a dramatic architectural leap. In many Java backends, it appears in a much smaller and more useful form.

If reads and writes already have different behavior, different performance requirements, and different models, it is often enough to separate query and command ports.

That is already a CQRS-style decision:

public interface SlotQueryRepository {
    List<PerformanceSlot> findPublishedSlots(SlotSearchCriteria criteria);
}

public interface SlotCommandRepository {
    void mergeSnapshot(List<PerformanceSlot> slots, Instant importedAt);
    void disableMissingSlots(Instant importedAt);
}

This is not full CQRS. There is no separate read database, no asynchronous projection system, and no event-driven choreography. But the separation is still valuable because it acknowledges that reads and writes have different semantics.

In practice, this buys you a lot:

the read side can optimize for filtering, sorting, pagination, and caching
the write side can optimize for consistency, idempotency, and synchronization logic
tests become more focused because the contracts are narrower

This is one of those places where I think pragmatism matters more than purity. You do not need the full CQRS stack to benefit from the idea.

SOLID and related patterns still matter

Hexagonal Architecture does not replace SOLID. It gives some of those principles a clearer architectural shape.

The most visible one is Dependency Inversion. Application services depend on abstractions such as PartnerCatalogClient or SlotQueryRepository, not on concrete adapters.

Single Responsibility also becomes easier to enforce when boundaries are explicit:

controllers deal with HTTP
schedulers trigger use cases
adapters handle translation
application services orchestrate
domain objects protect business rules

Clean Architecture and Onion Architecture belong to the same family of ideas. The naming differs, but the core intention is similar: keep business rules in the center and push technology outward.

The useful takeaway is not which label you choose. The useful takeaway is whether your code actually follows the dependency rule you claim to use.

Testing is where Hexagonal Architecture usually proves itself

The strongest argument for the pattern is rarely the diagram. It is the test suite.

A service designed around ports is much easier to test in layers:

domain tests for invariants
application service tests with mocked ports
adapter integration tests for persistence or external clients
controller tests for request and response mapping
architecture tests to enforce dependency rules

A unit test for the synchronization flow can stay fast and precise:

@ExtendWith(MockitoExtension.class)
class RefreshSlotCatalogServiceTest {

    @Mock
    private PartnerCatalogClient partnerCatalogClient;

    @Mock
    private SlotCommandRepository slotCommandRepository;

    @Mock
    private Clock clock;

    @InjectMocks
    private RefreshSlotCatalogService service;

    @Test
    void refreshUsesTheSameTimestampForMergeAndDisable() {
        Instant importedAt = Instant.parse("2026-01-01T00:00:00Z");
        when(clock.instant()).thenReturn(importedAt);
        when(partnerCatalogClient.fetchSlots()).thenReturn(List.of());

        service.refreshCatalog();

        verify(slotCommandRepository).mergeSnapshot(List.of(), importedAt);
        verify(slotCommandRepository).disableMissingSlots(importedAt);
    }
}

That test is quick, deterministic, and tied to business behavior instead of framework wiring.

I also strongly recommend architecture tests once the project has a few contributors. If the team says the domain must not depend on infrastructure, that rule should be executable:

@ArchTest
static final ArchRule domainMustNotDependOnInfrastructure = noClasses()
        .that().resideInAPackage("..domain..")
        .should().dependOnClassesThat()
        .resideInAnyPackage("..application..", "..infrastructure..");

Without those checks, many projects keep the folder names and lose the architecture.

Common mistakes that are worth avoiding early

There are a few mistakes I keep seeing in Java teams that adopt hexagonal structure too literally.

1. Ports everywhere

If a dependency is not a real boundary, do not force it into a port. A LoggerPort is almost always a smell. A ClockPort often is too. Architectural boundaries should represent things that may vary independently or that materially affect testability and coupling.

2. Anemic domain with fancy packaging

A domain package full of records or Lombok DTOs with no invariants is not a domain model. It is just data moved to a different folder.

3. Business logic hidden in adapters

As soon as adapters start deciding what is valid, what should be retried, or how core workflows should behave, the architecture is being bypassed.

4. Pretending folder names equal architecture

Folders help, but they do not enforce anything by themselves. Dependency rules, code review discipline, and a small amount of architectural testing do.

Conclusions

Hexagonal Architecture is not a silver bullet, and it is not the right answer for every application. For a small CRUD service with minimal business logic and few external dependencies, it might add more complexity than value. But for growing backends, especially in fast-moving industries like fintech, healthtech, or any domain with complex rules and frequent pivots, it is often the most solid long-term investment.

I've worked on systems where we've replaced entire infrastructure stacks, migrating from monoliths to microservices, switching message queues, or integrating new database systems, with minimal changes to core logic. That stability in the center while everything around it evolves is what makes the pattern powerful.

My advice: Start small and don't over-engineer. Apply the pattern to domains where volatility and growth risk are highest. If you're building a payment processing engine or a complex workflow system, hexagonal architecture makes sense from day one. If you're building a simple backend for a mobile app with CRUD operations, maybe start traditionally and refactor toward hexagonal boundaries as complexity grows. You can introduce ports and adapters incrementally, they're just interfaces and implementations, after all.

Self-Hosting n8n on Raspberry Pi: HTTPS, DDNS & Automated Backups

Juan Carlos González Cabrero — Thu, 19 Feb 2026 13:38:34 +0000

1. Why I Choose to Self-Host n8n on a Raspberry Pi

When I first started working with automation tools, n8n's cloud offering seemed like the best choice. But after a few months, I found myself increasingly concerned about where the data lives, especially when handling customer information and internal business logic. The costs were also scaling faster than I'd anticipated, and I kept running into those limitations that come with any SaaS platform: can't customize the infrastructure, can't control update timing, can't peek under the hood when something goes sideways.

As a backend engineer with curiosity and passion for challenges, the idea of self-hosting n8n on a Raspberry Pi started making more sense. I wanted complete control over my automation stack without the recurring charges or vendor dependencies. More importantly, I wanted to understand every layer of the system, from the basic setup to the reverse proxy to the backup strategy. This wasn't just about getting n8n running locally on a publicly exposed port and calling it done. I needed a setup that could handle production workloads, stay accessible 24/7 from anywhere, and recover gracefully from failures.

What I ended up building is what I'm sharing with you here: a fully Dockerized n8n stack with PostgreSQL, secure HTTPS endpoints, dynamic DNS to handle my home ISP's changing IP addresses, and automated cloud backups. It's been running reliably for some weeks now, handling everything from complex workflows to internal notification systems.

2. Stack Architecture and Core Configuration

This setup uses Docker Compose to isolate each component, n8n for workflow automation, PostgreSQL as the database, pgAdmin for the PostgreSQL management, and Nginx Proxy Manager for TLS, reverse proxy, and Let's Encrypt certificate management. Finally, the persistent volumes for each service are stored under ./data.

All sensitive values are stored in a single .env file in the project directory:

POSTGRES_USER=n8nprod
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_DB=n8n
PGADMIN_EMAIL=admin@example.com
PGADMIN_PASSWORD=change_me
N8N_BASIC_AUTH_ACTIVE=true
N8N_BASIC_AUTH_USER=automation
N8N_BASIC_AUTH_PASSWORD=change_me_later
N8N_ENCRYPTION_KEY=<32 char random string>
PUBLIC_DOMAIN=<YOUR_DOMAIN>
N8N_PROTOCOL=https
N8N_PORT=5678

Important: in .env, PUBLIC_DOMAIN=<YOUR_DOMAIN> must be the public domain you want to publish your n8n server, for example n8n.example.com.

The first thing I do after creating this .env file is lock down its permissions. Only the user running Docker should be able to read it. This prevents other users on the system (or compromised services) from accessing the secrets. You could use Docker secrets, or even HashiCorp Vault, but for a small setup, strict file permissions and regular password rotation are already a strong baseline:

chmod 600 .env

This is the Docker Compose file where everything comes together:

services:
  postgres:
    image: postgres:15
    container_name: postgres_db
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - db_network

  pgadmin:
    image: dpage/pgadmin4
    container_name: pgadmin
    restart: unless-stopped
    environment:
      PGADMIN_DEFAULT_EMAIL: ${PGADMIN_EMAIL}
      PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_PASSWORD}
      PGADMIN_CONFIG_SERVER_MODE: 'True'
      PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED: 'False'
    ports:
      - "8080:80"
    volumes:
      - ./data/pgadmin:/var/lib/pgadmin
    depends_on:
      - postgres
    networks:
      - db_network

  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    environment:
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: postgres
      DB_POSTGRESDB_PORT: 5432
      DB_POSTGRESDB_DATABASE: ${POSTGRES_DB}
      DB_POSTGRESDB_USER: ${POSTGRES_USER}
      DB_POSTGRESDB_PASSWORD: ${POSTGRES_PASSWORD}
      N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
      N8N_SECURE_COOKIE: true
      N8N_HOST: 0.0.0.0
      N8N_PORT: 5678
      N8N_PROTOCOL: https
      N8N_EDITOR_BASE_URL: https://${PUBLIC_DOMAIN}/
      WEBHOOK_URL: https://${PUBLIC_DOMAIN}/
    depends_on:
      - postgres
    volumes:
      - ./data/n8n:/home/node/.n8n
    networks:
      - db_network

  nginx:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    volumes:
      - ./data/npm/data:/data
      - ./data/npm/letsencrypt:/etc/letsencrypt
    depends_on:
      - n8n
      - pgadmin
    networks:
      - db_network

networks:
  db_network:
    driver: bridge

The volume mounts deserve special attention. For PostgreSQL, I'm mounting ./data/postgres to persist the database across container restarts. For n8n, the ./data/n8n mount stores workflow data and encrypted credentials. Nginx Proxy Manager stores config and certificates in ./data/npm, and pgAdmin persists its state in ./data/pgadmin. This path structure keeps backups simple because every critical state lives under ./data.

I use restart: unless-stopped for all services rather than restart: always. The distinction matters: if I manually stop a container for maintenance, I don't want it automatically restarting. But if the Pi reboots or Docker daemon restarts, everything should come back up without manual intervention.

After the first docker compose up -d, run a quick service sanity check:

docker compose ps
docker compose logs --tail=50 n8n
docker compose logs --tail=50 nginx

If n8n is restarting in loop, the first thing to verify is .env consistency, especially POSTGRES_*, N8N_ENCRYPTION_KEY, and PUBLIC_DOMAIN.

3. Exposing n8n Securely with Nginx Proxy Manager

Running n8n directly on port 5678 with HTTP is fine for localhost testing, but completely unacceptable for production use. You need HTTPS for security, you need proper hostname routing, and you need automatic certificate renewal. I used to configure all of this manually with raw nginx configs and certbot, but Nginx Proxy Manager (NPM) has simplified my life considerably.

NPM provides a clean web UI, usually accessible on port 81, where you can configure proxy hosts, SSL certificates, and access controls without touching nginx config files.

The SSL configuration is remarkably straightforward. In Nginx Proxy Manager, create a proxy host for <YOUR_DOMAIN>, forward to n8n:5678, enable Force SSL, and request a Let's Encrypt certificate. NPM handles the ACME challenge automatically, as long as ports 80 and 443 are properly forwarded from your router. The certificates renew automatically every 60 days, and I've never had to intervene manually. This is the kind of automation that makes self-hosting sustainable.

What I particularly appreciate about this architecture is that all TLS termination happens at the proxy layer. n8n itself runs plain HTTP internally, which simplifies its configuration and reduces the attack surface. Only NPM needs access to private keys, and only NPM faces the public internet directly. This separation of concerns makes security auditing much easier.

4. Domain config with Dynamic DNS (DDNS) and Router Setup

Domain and DDNS flow:

Register domain at your registrar.
Delegate nameservers to Cloudflare.
Create DNS record for your n8n host if needed.

In my case I created an A record for a subdomain I reserved for my n8n instance, and that record is the one my DDNS process updates automatically.
Keep DNS updated with ddns-updater.

Install DDNS updater:

go install github.com/qdm12/ddns-updater/cmd/ddns-updater@latest
mkdir -p ~/ddns/data
mv ~/go/bin/ddns-updater ~/ddns/

This is the configuration file config.json you need to create under ~/ddns/data directory, following the tool documentation:

{
  "settings": [
    {
      "provider": "cloudflare",
      "domain": "<YOUR_DOMAIN>",
      "token": "<CLOUDFLARE_API_TOKEN>",
      "zone_identifier": "<CLOUDFLARE_ZONE_ID>"
    }
  ]
}

To keep the updater alive 24/7, I created a systemd service:

# /etc/systemd/system/ddns-updater.service
[Unit]
Description=DDNS Updater
After=network-online.target
Wants=network-online.target

[Service]
User=<your_user>
WorkingDirectory=/home/<your_user>/ddns
ExecStart=/home/<your_user>/ddns/ddns-updater -datadir ./data
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now ddns-updater.service
systemctl status ddns-updater.service
journalctl -u ddns-updater.service -f

Your router needs two settings, a fixed local IP for the Raspberry Pi, and port forwarding for 80 and 443 to that same IP.

First, get the network interface MAC address from the Raspberry Pi:

ifconfig -a

Then, in your router DHCP section, bind that MAC to a fixed LAN IP you choose. It can be any valid free IP in your subnet, for example 192.168.1.100.

Finally, create port forwarding rules pointing to that same fixed IP:

TCP 80 -> <fixed_lan_ip>:80
TCP 443 -> <fixed_lan_ip>:443

If the forwarded IP and the DHCP reservation IP do not match, HTTPS issuance and public access will fail.

Once DNS and forwarding are configured, validate the public path:

dig +short <YOUR_DOMAIN> @1.1.1.1
curl -I https://<YOUR_DOMAIN>

Expected result, dig returns your public IP (or Cloudflare edge IPs if proxied), and curl returns an HTTP response over TLS.

5. Automated Backups with rclone

Backups include both PostgreSQL data and n8n runtime data (data/n8n). rclone is reliable for unattended uploads, supports retries, and keeps the process lightweight on Raspberry Pi.

My backup script runs nightly via cron. It loads .env, dumps the database from the running PostgreSQL container, archives n8n data, uploads both files to a timestamped Google Drive folder, and enforces local and remote retention policies.

#!/usr/bin/env bash
set -euo pipefail

PROJECT_DIR="$(pwd)"
ENV_FILE="$PROJECT_DIR/.env"
BACKUP_DIR="$HOME/.n8n/backups"
RCLONE_REMOTE="gdrive:n8n-backups"
LOCAL_RETENTION_DAYS=14
REMOTE_RETENTION_DAYS=90
TS=$(date +"%Y-%m-%d_%H-%M-%S")

set -a
source "$ENV_FILE"
set +a

PG_DUMP_FILE="$BACKUP_DIR/postgres_${TS}.sql"
N8N_TAR_FILE="$BACKUP_DIR/n8n_data_${TS}.tar.gz"
mkdir -p "$BACKUP_DIR"

docker compose exec -T postgres \
  pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" \
  > "$PG_DUMP_FILE"

tar -czf "$N8N_TAR_FILE" -C "$PROJECT_DIR" data/n8n

REMOTE_DIR="$RCLONE_REMOTE/$TS"
rclone mkdir "$REMOTE_DIR"
rclone copy "$PG_DUMP_FILE" "$REMOTE_DIR"
rclone copy "$N8N_TAR_FILE" "$REMOTE_DIR"

find "$BACKUP_DIR" -type f -mtime +"$LOCAL_RETENTION_DAYS" -delete
rclone delete "$RCLONE_REMOTE" --min-age "${REMOTE_RETENTION_DAYS}d" --rmdirs

echo "Backup OK: $TS"

Reliability depends on automation that does not require manual intervention. I use cron for backups, and a long running systemd service for DDNS.

This backup script runs via cron at 3 AM daily. The cron entry pipes output to a log file, so I have a record of every backup attempt.

0 3 * * * cd /home/pi/n8n-server && /bin/bash ./n8n-backup.sh >> /var/log/n8n-backup.log 2>&1

Operational check, verify new backup folders are being created in Google Drive and validate local log history:

rclone lsd gdrive:n8n-backups | tail -n 5
tail -n 50 /var/log/n8n-backup.log

For a quick restore drill, create a temporary database and import one dump:

set -a && source .env && set +a
docker compose exec -T postgres psql -U "$POSTGRES_USER" -c 'CREATE DATABASE n8n_restore_test;'
docker compose exec -T postgres psql -U "$POSTGRES_USER" n8n_restore_test < "$HOME/.n8n/backups/postgres_YYYY-MM-DD_HH-MM-SS.sql"

If this works and n8n data archive can be extracted without errors, your backup pipeline is operational, not just successful on paper.

6. Security and Operations Baseline

Minimal baseline for a n8n service facing public exposure:

Restrict inbound ports with UFW firewall, allowing only 80, 443, and local SSH.

sudo ufw allow 80,443/tcp
sudo ufw allow from 192.168.1.0/24 proto tcp to any port 22
sudo ufw enable

Enforce HTTPS end to end at the public boundary.

Nginx Proxy Manager should have Force SSL enabled for the n8n host, and n8n must keep secure URL settings:

N8N_PROTOCOL=https
N8N_SECURE_COOKIE=true
N8N_EDITOR_BASE_URL=https://${PUBLIC_DOMAIN}/
WEBHOOK_URL=https://${PUBLIC_DOMAIN}/

This avoids mixed HTTP/HTTPS behavior and ensures cookies and webhook callbacks stay in secure mode.

Rotate credentials and minimize secret exposure.

Rotate n8n basic auth credentials, PostgreSQL credentials, and Cloudflare API token on a fixed schedule. Use a Cloudflare token scoped only to the required zone and DNS permissions, never your global API key. Keep .env and DDNS config files outside any version control system.

Most critical detail for n8n recovery, keep N8N_ENCRYPTION_KEY backed up in a secure location. Without that key, stored n8n credentials cannot be decrypted after restore.

Harden SSH access.

Disable password authentication and root login, use SSH keys only, and keep SSH reachable only from your private network range:

sudo sed -i 's/^#\\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Validate backup restorations, not only backup creation.

A successful backup command does not guarantee a valid restore. Test restores periodically in an isolated environment and verify:

PostgreSQL dump can be restored without errors.
data/n8n archive is complete.
n8n boots correctly with the original N8N_ENCRYPTION_KEY.
Critical workflows and credentials load correctly.

Keep management surfaces private.

Do not expose admin ports publicly. In practice, only forward 80 and 443 in your router. Keep 81 (NPM admin), 8080 (pgAdmin), and 5678 (direct n8n) private to LAN or behind VPN.

A simple verification is checking exposed listeners from the host:

sudo ss -tulpen | grep -E '(:80|:81|:443|:5678|:8080)\\b'

Public exposure should be limited to 80/443 through your router rules.

If everything is configured correctly, opening https://<YOUR_DOMAIN> should display the n8n login page. This is the expected result after completing the setup and routing public traffic from your domain to the Raspberry Pi server.

7. Conclusions and Lessons Learned

This setup covers the full path from local deployment to stable public exposure, Docker based service isolation, HTTPS termination, DDNS automation, router alignment with DHCP reservation, and backup automation with restore oriented validation.

The most important lessons were operational, not tooling specific:

Reliability comes from consistent automation, cron, systemd, and health checks.
Security comes from reducing exposed surface, strict secret handling, and access hardening.
Backups are useful only when restore is tested and repeatable.
Portability improves when domain, credentials, and runtime paths are centralized in environment variables.
The same architecture can be reproduced on Raspberry Pi, or cloud VMs with minimal changes.

If you follow these controls from day one, you get a setup that is easier to operate, easier to recover, and safer to expose publicly.

API-First with OpenAPI Generator: From Spec to REST API and Type-Safe SDKs

Juan Carlos González Cabrero — Thu, 22 Jan 2026 13:23:37 +0000

1. Introduction & Contract-First Approach

APIs are the arteries of modern software, powering everything from mobile apps to distributed cloud microservices. Building those APIs, however, is rarely as straightforward as writing a few controller methods. If you've worked on any enough complex backend, you've probably wrestled with inconsistent request/response payloads, mismatched client/server contracts, or ambiguous endpoint documentation. I certainly have, and the pain points are always the same: a frontend developer working from outdated documentation, a client blocked because the API shape changed unexpectedly, or integration tests failing because nobody synchronized the contract.

That's why I advocate for the "contract-first," or API-first, approach. Instead of treating the OpenAPI specification as an afterthought—something you generate from annotations or write to satisfy a documentation requirement—you define your API contract before implementing it. This inverts the traditional workflow in a way that fundamentally changes how teams collaborate. Your OpenAPI spec becomes the single source of truth that both server and client implementations derive from, ensuring they can never drift apart. The spec drives automatic, always-current documentation. It aligns product managers, frontend engineers, backend developers, and external partners around a shared understanding before a single line of implementation code is written.

With tools like the OpenAPI Generator and Maven, you can turn a single OpenAPI spec into a production-grade Java REST backend and type-safe SDK clients in multiple languages. I've seen this approach cut integration time from weeks to days in microservice architectures and eliminate entire categories of bugs related to contract mismatches. Today, I'll walk you through building a real workflow for this: from designing the OpenAPI 3.0 spec, to generating Spring Boot controllers and Java client SDKs, to handling API evolution gracefully. I'll also share the gotchas I've learned the hard way in production environments.

2. OpenAPI Specification Setup

The foundation of everything we're building starts with the OpenAPI spec itself. In a contract-first workflow, your OpenAPI YAML or JSON file isn't just documentation—it's an executable contract that drives code generation, testing, and deployment. Every detail matters because ambiguity in the spec translates directly to ambiguity in generated code. I've learned to be obsessively precise here, because time invested in a well-crafted spec pays exponential dividends downstream.

Consider a basic bookstore API. Before writing any Java controllers or setting up Spring Boot, we define exactly how this API should behave. Here's what that looks like in OpenAPI 3.0:

openapi: 3.0.3
info:
  title: Bookstore API
  version: 1.0.0
  description: |-
    APIs to manage books and orders in a bookstore.
servers:
  - url: http://localhost:8080/api
paths:
  /books:
    get:
      summary: List all books
      responses:
        '200':
          description: A list of books
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Book'
    post:
      summary: Add a new book
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/BookRequest'
      responses:
        '201':
          description: Book created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Book'
components:
  schemas:
    Book:
      type: object
      required:
        - id
        - title
        - author
      properties:
        id:
          type: string
          format: uuid
        title:
          type: string
        author:
          type: string
        price:
          type: number
        inStock:
          type: boolean
    BookRequest:
      type: object
      required:
        - title
        - author
      properties:
        title:
          type: string
          minLength: 1
        author:
          type: string
        price:
          type: number
          minimum: 1
        inStock:
          type: boolean
          default: true

Notice the level of specificity. I'm not just saying "there's a title field"—I'm declaring it's required, it's a string, and it must have at least one character. The price must be a number with a minimum value of one. The Book model requires an ID, title, and author, while the BookRequest model (what clients send when creating a book) has slightly different requirements. This separation between request and response models is deliberate. In production, you rarely want clients sending IDs for new resources—those should be server-generated. By defining distinct schemas, the generated code enforces these business rules at compile time.

The validation constraints embedded here—minLength, minimum, required—aren't just documentation. When we generate code in the next section, these become actual Hibernate Validator annotations in your Java models. Client SDKs gain the same type safety. A TypeScript client will know that price is a number, not a string. A Go client will have required fields that can't be omitted. This is the power of treating the contract as code: you're not just describing the API, you're programming the behavior of every system that interacts with it.

For project organization, I recommend keeping your OpenAPI spec in the resources directory in your repository. In a Maven project, the structure typically looks like this:

project-root/
  src/
    main/
      java/...
      resources/
        bookstore-api.yaml
        ...
    test/
      java/...
  pom.xml

This separation also enables smooth integration with CI/CD tools. Your pipeline can lint the spec, generate documentation, run contract tests, and publish versioned artifacts—all independent of your Java compilation. The spec becomes the input to multiple downstream processes, not just a sidecar to your server implementation.

3. Server-Side Generation

With a solid spec in hand, we're ready to generate our Spring Boot server code. The OpenAPI Generator Maven plugin is the engine that transforms our declarative YAML into concrete Java interfaces and model classes. Configuring this plugin correctly is crucial because it determines not just what code gets generated, but how that code integrates with your handwritten business logic.

Here's the plugin configuration in your pom.xml that I've refined across multiple production projects:

<plugin>
  <groupId>org.openapitools</groupId>
  <artifactId>openapi-generator-maven-plugin</artifactId>
  <version>7.12.0</version>
  <executions>
    <execution>
      <id>code-generate-spring</id>
      <goals>
        <goal>generate</goal>
      </goals>
      <configuration>
        <inputSpec>${project.basedir}/src/main/resources/bookstore-api.yaml</inputSpec>
        <generatorName>spring</generatorName>
        <apiPackage>com.bookstore.api</apiPackage>
        <modelPackage>com.bookstore.model</modelPackage>
        <invokerPackage>com.bookstore.invoker</invokerPackage>
        <library>spring-boot</library>
        <configOptions>
          <reactive>true</reactive>
          <delegatePattern>true</delegatePattern>
          <interfaceOnly>true</interfaceOnly>
          <useTags>true</useTags>
          <dateLibrary>java8</dateLibrary>
          <useBeanValidation>true</useBeanValidation>
        </configOptions>
      </configuration>
    </execution>
  </executions>
</plugin>

The interfaceOnly option is the key architectural decision here. When set to true, the generator creates Java interfaces for each API path—not concrete implementations. This gives you the perfect separation of concerns: the generated code defines the contract (method signatures, parameter types, return types), while your handwritten code provides the implementation. I cannot overstate how valuable this separation becomes as APIs evolve. When you modify your OpenAPI spec and regenerate, your IDE immediately shows you which controller methods need updates because the interface changed. No grep'ing through logs, no runtime surprises—just compile-time feedback.

The useBeanValidation flag enables Hibernate Validator annotations on generated model classes. Remember those minLength and minimum constraints in our spec? They become @NotNull, @Min, and @Size annotations in the generated Java code. Spring Boot's validation framework automatically enforces these when requests arrive, rejecting invalid data before it reaches your business logic.

After running mvn clean generate-sources, you'll find generated code under target/generated-sources/openapi. I've seen teams make the mistake of modifying this generated code directly. Don't. Treat generated sources as immutable build artifacts, like compiled .class files. Any changes you make will be overwritten on the next build. Instead, your implementations live in your main source directory and reference the generated interfaces:

package com.bookstore.controller;

import com.bookstore.api.BooksApi;
import com.bookstore.model.Book;
import com.bookstore.model.BookRequest;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
public class BooksController implements BooksApi {

    private final BookRepository bookRepository;
    private final BookService bookService;

    public BooksController(BookRepository bookRepository, BookService bookService) {
        this.bookRepository = bookRepository;
        this.bookService = bookService;
    }

    @Override
    public ResponseEntity<List<Book>> getBooks() {
        List<Book> books = bookRepository.findAll();
        return ResponseEntity.ok(books);
    }

    @Override
    public ResponseEntity<Book> addBook(BookRequest bookRequest) {
        Book createdBook = bookService.addBook(bookRequest);
        return ResponseEntity.status(201).body(createdBook);
    }
}

This pattern feels natural in practice. You're implementing an interface, just like any other Java development. The difference is that this interface was derived from your API contract, so you get compile-time guarantees about correctness. If you add a new required parameter to an endpoint in your OpenAPI spec, the generated interface changes, and your controller won't compile until you update the implementation. This catches integration bugs at build time rather than in staging or—worse—production.

The validation annotations on generated models work automatically with Spring's @Valid annotation, but surfacing errors to clients in a user-friendly format requires a bit of plumbing. In production, I always implement a global exception handler to transform validation failures into structured error responses:

@ControllerAdvice
public class ApiExceptionHandler {

    @ExceptionHandler(MethodArgumentNotValidException.class)
    public ResponseEntity<ApiError> handleValidationError(MethodArgumentNotValidException ex) {
        ApiError error = new ApiError();
        error.setStatus(400);
        error.setMessage("Validation failed");
        error.setErrors(ex.getBindingResult()
            .getFieldErrors()
            .stream()
            .map(err -> err.getField() + ": " + err.getDefaultMessage())
            .collect(Collectors.toList()));

        return ResponseEntity.badRequest().body(error);
    }
}

This ensures that when a client sends a book with a negative price or an empty title, they receive a clear, actionable error message rather than a generic 500 or a stack trace. The validation rules specified once in your OpenAPI contract now flow all the way through to the error responses your clients see.

One pitfall I've encountered repeatedly: teams sometimes struggle with the generated code being "in the way" during development. They're tempted to edit it for quick fixes or to add custom annotations. The solution is to use extension points. If you need custom behavior, extend or wrap the generated classes in your own source tree. Use .openapi-generator-ignore to prevent the generator from overwriting specific files if you absolutely must customize generated code, but be cautious—you're opting out of automatic contract enforcement for those files. In most cases, composition beats modification: write adapter classes that delegate to generated code while adding your customizations.

4. Client SDK Generation

The contract-first approach truly shines when you realize your OpenAPI spec can generate not just server code, but type-safe client SDKs in virtually any language. This is transformative for microservice architectures and external API consumers. Instead of each client team hand-rolling HTTP requests and parsing JSON, they get a strongly-typed SDK that's automatically synchronized with your server implementation—because both derive from the same contract.

Generating a Java client SDK uses the same plugin with a different generator configuration:

<execution>
  <id>generate-java-client</id>
  <goals>
    <goal>generate</goal>
  </goals>
  <configuration>
    <inputSpec>${project.basedir}/src/main/resources/bookstore-api.yaml</inputSpec>
    <generatorName>java</generatorName>
    <library>resttemplate</library>
    <apiPackage>com.bookstore.client.api</apiPackage>
    <modelPackage>com.bookstore.client.model</modelPackage>
    <output>${project.build.directory}/generated-sources/java-client</output>
  </configuration>
</execution>

After running mvn clean generate-sources, you have a fully functional Java client SDK with the same strong typing as your server. Here's what using that SDK looks like in a separate microservice or integration test:

import com.bookstore.client.api.BooksApi;
import com.bookstore.client.model.Book;
import com.bookstore.client.model.BookRequest;
import org.springframework.web.client.RestTemplate;

public class BookInventoryFetcher {
    private final BooksApi booksApi;

    public BookInventoryFetcher(String apiUrl) {
        RestTemplate restTemplate = new RestTemplate();
        booksApi = new BooksApi();
        booksApi.setApiClient(new ApiClient(restTemplate).setBasePath(apiUrl));
    }

    public List<Book> fetchBooks() {
        return booksApi.getBooks();
    }

    public Book createBook(String title, String author, Double price) {
        BookRequest request = new BookRequest();
        request.setTitle(title);
        request.setAuthor(author);
        request.setPrice(price);

        return booksApi.addBook(request);
    }
}

Notice what's happening here. The client code has no raw HTTP calls, no JSON parsing, no string concatenation of URLs. The BooksApi class provides type-safe methods like getBooks() that return List<Book>. The Book and BookRequest models have the same fields, types, and validation constraints as the server. If the server API changes—say, you rename inStock to available—the client SDK regenerates with that change, and any code using the old field name stops compiling. This is contract enforcement at its finest.

The same workflow applies to other languages. Want a TypeScript client for your web frontend? Change generatorName to typescript-axios. Need a Python SDK for a data pipeline? Use python. Go for infrastructure tooling? go. The OpenAPI Generator supports dozens of languages, each with multiple library options. In one project, I maintained server code in Java, a TypeScript SDK for the React frontend, a Python SDK for ML engineers, and a Go SDK for infrastructure automation—all generated from the same OpenAPI spec. When the API evolved, I regenerated all four SDKs in a single build step. The alternative—manually maintaining four client implementations—would have been a coordination nightmare.

Publishing these SDKs is straightforward. Package the Java client as a Maven artifact and deploy it to your internal repository. Use npm for TypeScript, PyPI for Python, and so on. Version the SDKs to match your API version, and consuming teams can depend on them like any other library. This transforms your internal APIs into first-class, versioned products rather than endpoints that teams access via curl and prayer.

5. Advanced Patterns

In production, APIs don't stay static. Requirements change, new features emerge, and you discover design flaws that need correction. The contract-first approach, combined with generated code, provides elegant patterns for handling this evolution while maintaining stability for existing consumers.

Maintaining multiple API versions side-by-side is surprisingly straightforward. Keep separate spec files—bookstore-v1.yaml and bookstore-v2.yaml—and configure separate plugin executions for each. Your server can implement both BooksApiV1 and BooksApiV2 interfaces, routing requests based on a version header or URL path segment like /api/v1/books versus /api/v2/books. I've used this pattern to keep legacy endpoints alive for mobile apps that can't update immediately while rolling out breaking changes to web clients. The generated code keeps each version's contract enforced independently, preventing accidental cross-contamination of v1 and v2 logic.

Integrating code generation into your CI/CD pipeline automates contract enforcement across your entire development workflow. In GitHub Actions or Jenkins, add a build step that runs mvn clean generate-sources and fails if generated code doesn't match what's committed. This catches developers who modified the spec locally but forgot to regenerate code, or vice versa. Here's a snippet from a GitHub Actions workflow that I use:

name: API Contract Validation
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          java-version: '17'
      - name: Generate OpenAPI code
        run: mvn clean generate-sources
      - name: Check for uncommitted changes
        run: |
          git diff --exit-code target/generated-sources/openapi

If generated code differs from what's in the repository, the build fails. This simple check has saved me from countless "it works on my machine" bugs related to contract drift.

Contract testing takes this further by validating that both server and client implementations actually conform to the spec, not just that code was generated from it. Tools like Schemathesis or Spring Cloud Contract can execute tests against the running server using the OpenAPI spec as the test definition. These tests send requests covering every endpoint and parameter combination defined in the spec, then validate responses match the schema. I've caught bugs where business logic returned nullable fields that the spec declared non-nullable, or where enum values in practice diverged from the contract. Traditional unit tests rarely catch these issues because they test specific scenarios, not the entire contract surface.

Documentation generation is almost trivial when your API is defined in OpenAPI. Adding Springdoc-OpenAPI to your dependencies automatically serves interactive Swagger UI documentation:

springdoc.api-docs.path=/api-docs
springdoc.swagger-ui.path=/swagger-ui.html

With the server running, navigate to /swagger-ui.html and you have live, interactive API documentation that stays in sync with your implementation because it's reading directly from the runtime application. For static documentation—say, for embedding in a developer portal—generate ReDoc HTML as part of your build:

npx redoc-cli bundle src/main/resources/bookstore-api.yaml -o docs/api-documentation.html

I've seen teams struggle with documentation staleness for years before adopting this approach. Once your spec is the source of truth, documentation becomes a free byproduct rather than a maintenance burden.

6. Real-World Example: Bookstore Microservice

To see how all these pieces fit together in practice, consider a microservices architecture where Product and Inventory services need to coordinate. The Product service manages book metadata and handles customer queries. The Inventory service tracks stock levels and processes reservations. Both need to share a common understanding of what a "book" is and how stock checks work.

Start by defining a shared OpenAPI contract in bookstore-api.yaml that includes endpoints for querying available books and reserving inventory. In the Product service's pom.xml, configure the generator to create Spring server interfaces:

<execution>
  <id>code-generate-spring</id>
  <goals><goal>generate</goal></goals>
  <configuration>
    <inputSpec>${project.basedir}/src/main/resources/bookstore-api.yaml</inputSpec>
    <generatorName>spring</generatorName>
    <apiPackage>com.company.bookstore.product.api</apiPackage>
    <modelPackage>com.company.bookstore.model</modelPackage>
    <configOptions>
      <interfaceOnly>true</interfaceOnly>
      <useBeanValidation>true</useBeanValidation>
    </configOptions>
  </configuration>
</execution>

The Product service implements these interfaces to expose REST endpoints. Meanwhile, in the Inventory service's pom.xml, configure the generator to create a Java client:

<execution>
  <id>generate-java-client</id>
  <goals><goal>generate</goal></goals>
  <configuration>
    <inputSpec>${project.basedir}/src/main/resources/bookstore-api.yaml</inputSpec>
    <generatorName>java</generatorName>
    <library>resttemplate</library>
    <apiPackage>com.company.bookstore.inventory.client</apiPackage>
    <modelPackage>com.company.bookstore.model</modelPackage>
    <output>${project.build.directory}/generated-sources/java-client</output>
  </configuration>
</execution>

Now the Inventory service can call Product service endpoints using a type-safe SDK instead of raw HTTP. Both services share identical model definitions—the Book class is generated identically in both codebases because it comes from the same spec. If you add a new field to Book in the contract, both services regenerate with that field. The Product service's endpoints automatically accept and return the new field, and the Inventory service's client SDK includes it. There's no way for the services to drift apart because the contract binds them together.

For frontend integration, add a TypeScript client generation step:

<execution>
  <id>generate-typescript-client</id>
  <goals><goal>generate</goal></goals>
  <configuration>
    <inputSpec>${project.basedir}/src/main/resources/bookstore-api.yaml</inputSpec>
    <generatorName>typescript-axios</generatorName>
    <output>${project.build.directory}/generated-sources/typescript-client</output>
  </configuration>
</execution>

Package the TypeScript SDK as an npm module, publish it to your registry, and frontend developers import it like any library. When they call booksApi.getBooks(), TypeScript's compiler enforces that they handle the response correctly—it knows the shape of the Book object, which fields are optional, and what types they have. The entire stack, from database to UI, is now bound by a single contract.

In practice, I've seen this approach cut feature development time dramatically. A new "add review" feature that touches frontend, Product service, and Inventory service used to require careful coordination between three teams to ensure everyone's JSON payloads matched. With generated code, we updated the OpenAPI spec with a new /books/{id}/reviews endpoint, regenerated all artifacts, and each team implemented against their generated interfaces. Integration happened in hours, not days, because mismatches were caught at compile time.

7. Pitfalls & Lessons Learned

I've deployed this pattern across fintech platforms, e-commerce marketplaces, and SaaS products. It's not a silver bullet—there are scenarios where it struggles and mistakes that can undermine the benefits. Here's what I've learned the hard way.

Code generation makes the most sense when your API contract is relatively stable. If your spec is in flux—changing daily as you explore a new feature space—you'll spend more time regenerating and updating implementations than you save. In early-stage projects or when prototyping, I sometimes write controllers by hand first, iterate until the design feels right, then reverse-engineer an OpenAPI spec and switch to generation for the production implementation. Trying to spec-first before you understand the problem space leads to churn and frustration.

Breaking changes are the eternal challenge of API evolution. Even with perfect tooling, removing a field or changing a data type breaks clients. Versioning is the answer, but it requires discipline. When introducing breaking changes, bump the API version in your spec (e.g., from 1.0.0 to 2.0.0), generate new endpoints under /v2, and keep /v1 alive until all consumers migrate. Set deprecation timelines and communicate them clearly. The generated code can help here—run both v1 and v2 servers simultaneously in the same Spring Boot app, each with their own generated interfaces, until v1 traffic drops to zero.

Managing generated code across repository boundaries requires careful dependency management. If you're generating and publishing client SDKs, version them carefully. Pin SDK versions in consuming projects to avoid surprise breakages. I've seen teams publish SDK patches that silently changed behavior because they forgot to bump the version number. Treat SDKs as public APIs themselves, with semantic versioning and changelogs.

Never, ever modify generated source files directly. It's tempting when you're in a hurry—just add that one annotation, tweak that method signature—but you've now created a time bomb. The next developer runs a build, the generator overwrites your change, and hours of debugging ensue. Use .openapi-generator-ignore sparingly and only for files you truly want to manage manually, like README files or example configurations. For code customizations, extend generated classes or use adapter patterns in your source tree.

Dependency conflicts between generated code and your application can be subtle. The generated client might pull in a version of Jackson or Spring that conflicts with your main application. Manage this with dependency exclusions in your POM and careful choice of generator libraries. I typically generate clients in separate Maven modules to isolate their dependencies from the main application.

Test automation is non-negotiable. Just because code is generated doesn't mean it's correct. Your OpenAPI spec might have a typo, or your business logic might not match the contract. Write integration tests that exercise the full request/response cycle. Run contract tests that validate the server against the spec. Test client SDKs against the running server. The tooling gives you compile-time safety for the interface, but runtime correctness is still your responsibility.

8. Conclusions and Final Thoughts

The contract-first approach—defining your OpenAPI spec before writing code—fundamentally changes how teams build and maintain APIs. By using OpenAPI Generator and Maven, you transform a declarative YAML file into a comprehensive suite of server interfaces, data models, client SDKs, and documentation. This inversion of the traditional workflow has profound effects: integration issues surface at compile time rather than runtime, client and server implementations can't drift apart, and documentation stays current because it's generated from the same source of truth.

From real-world experience across multiple production systems, the productivity gains are substantial. Features that once required careful coordination between backend and frontend teams—with inevitable integration delays when reality didn't match assumptions—now proceed in parallel with confidence. The contract provides a fence that both sides can develop against independently, meeting in the middle with generated code ensuring compatibility.

The approach scales particularly well as systems grow. Adding a new microservice that needs to call existing APIs? Generate a client SDK and you're working with type-safe methods, not HTTP libraries and JSON parsing. Exposing APIs to external partners? Publish SDKs in their language of choice, all guaranteed to match your server implementation. Supporting mobile apps that update slowly? Keep old API versions alive with separate generated interfaces until usage drops to zero.

The benefits compound with API maturity. Initial setup has friction—learning the generator options, deciding on project structure, establishing workflows—but once established, the patterns become second nature. Each new endpoint added to the spec automatically propagates to server code, client SDKs, and documentation. Each field added to a model updates everywhere simultaneously. The maintenance burden of keeping disparate systems synchronized largely disappears.

This isn't a magic solution to all API challenges. You still need to design good APIs, with sensible resource models and clear semantics. You still need to manage versions and communicate breaking changes. You still need comprehensive testing. But the contract-first approach with code generation removes an entire category of problems—the tedious, error-prone work of keeping implementations synchronized with contracts. It lets you focus on the hard problems: what the API should do, not whether all the pieces agree on how it's supposed to work.

For teams building distributed systems, microservices, or public APIs, the investment in contract-first development with OpenAPI Generator pays off quickly. The upfront effort to learn the tooling and establish patterns is measured in days. The ongoing benefits—fewer integration bugs, faster development, automatic documentation, type-safe clients—accrue over the entire lifetime of the API. In an industry where API maintenance is a major cost driver, this approach provides rare leverage: do the work once in the contract, and propagate it automatically everywhere it's needed.

Building a Cloud-Native SaaS Backend on GCP

Juan Carlos González Cabrero — Sat, 20 Dec 2025 20:31:05 +0000

Introduction: The Invisible Engine Behind Modern SaaS

When a user clicks 'Sign Up' on a SaaS product or requests some specific data, they expect real-time responsiveness and reliability. Behind this simple interaction runs a sophisticated backend, architected to scale, self-heal, and distribute load across a constellation of microservices. But as more startups embrace cloud-native designs and containerized services become the backbone, one challenge repeatedly emerges: how can we intelligently balance traffic so that it's not just spread evenly, but routed to the healthiest, fastest, and most reliable service endpoints?

This is far more complex than classic round-robin routing. As anyone running production systems has learned, naive traffic distribution leads to cascading failures when one service goes unhealthy, or bottlenecks when new versions aren't production-ready. In this article, I'll share a detailed backend architecture for cloud-native SaaS on GCP, focusing on intelligent load balancing for Dockerized Python microservices—using Cloud Load Balancing, GKE/Cloud Run, managed VPC, robust IAM, and native observability features.

1. Problem Context: Why Naive Load Balancing Fails in Production

Picture your SaaS backend composed of User, Billing, and Notification microservices, each containerized with Python and running in GKE. Your API Gateway distributes traffic through Cloud Load Balancer to whichever pods are registered. Everything looks fine in staging. Then production happens.

A new Billing pod version deploys that takes 30 seconds to warm up its database connection pool. Or perhaps a pod gets bogged down handling a batch export task, spiking latency to 5x normal. Maybe there's a memory leak that slowly degrades performance over hours. Classic load balancers will continue routing users to these struggling pods because, technically, they're still responding to basic health checks. The result? Your P95 latency climbs, timeout errors cascade through dependent services, and customer support tickets flood in.

Even with built-in Kubernetes readiness probes, the default GCP-managed load balancer doesn't always have granular-enough health data to avoid slow or failing endpoints instantly. The probe might check every 10 seconds, but a pod can fail spectacularly in the intervening time. What we need is intelligent load balancing driven by detailed health signals, readiness gates, real-time metrics, and rapid failure detection. The architecture I'm about to walk you through addresses exactly these challenges, drawn from years of running production SaaS platforms on Google Cloud.

2. Defining Intelligent Load Balancing: Key Requirements

Before writing a single line of code or provisioning any infrastructure, I've learned it's critical to be precise about what 'intelligent' actually means in this context. Too often, teams jump straight to implementation without defining success criteria, only to discover months later that their load balancing strategy has subtle but critical gaps.

Intelligent load balancing means the system only sends traffic to pods that are healthy, live, and genuinely responsive—not just pods that haven't crashed yet. It means distinguishing between containers that are technically running and those that are actually ready to handle production traffic. I've seen too many incidents where a pod passes its health check but is still initializing its database connections or warming up caches, leading to timeouts for the first users who hit it.

Beyond simple health, intelligent routing must consider real-time performance characteristics. A pod might be healthy but currently experiencing high latency due to garbage collection or resource contention. The load balancer should prefer endpoints with lower, more stable response times. When a pod starts showing elevated error rates or slowdowns, the system needs a feedback loop to temporarily route around it, even if traditional health checks still show it as operational.

The architecture also needs to play nicely with elastic scaling. As pods spin up and down in response to traffic patterns, the load balancer must smoothly integrate new capacity while draining traffic from pods scheduled for termination. And critically, all of this needs observability built in from day one. Without logs, traces, and metrics feeding back into routing decisions, you're flying blind. This is where GCP's integrated tooling becomes invaluable, providing the telemetry foundation that makes intelligent decisions possible.

3. Designing the Cloud-Native Backend Architecture

3.1 Microservices Design (Python, Docker)

The foundation of intelligent load balancing starts with services that properly communicate their state. I've found that too many microservices treat health checks as an afterthought, implementing them with a simple "return 200 OK" that tells the load balancer nothing useful. Instead, your services need to expose granular information about their actual readiness and health.

Here's a Python-based billing service that demonstrates the pattern I use in production. Notice how it separates health (is the process alive?) from readiness (is it prepared to serve traffic?):

# billing_service.py
from flask import Flask, jsonify
import random
import time

app = Flask(__name__)

@app.route("/healthz")
def health():
    # Report healthy 95% of the time, failure 5%
    if random.random() < 0.95:
        return "OK", 200
    else:
        return "Unhealthy", 500

@app.route("/readyz")
def ready():
    # Simulate readiness delay on startup
    if time.time() - START_TIME < 10:
        return "Not Ready", 503
    return "Ready", 200

@app.route("/pay", methods=["POST"])
def pay():
    # Simulate payment processing latency
    latency = random.uniform(0.05, 1.5)
    time.sleep(latency)
    return jsonify({"status": "success", "latency": latency})

if __name__ == "__main__":
    global START_TIME
    START_TIME = time.time()
    app.run(host='0.0.0.0', port=8080)

This separation between /healthz and /readyz mirrors what I've implemented across dozens of production services. The health endpoint tells Kubernetes whether the process should be restarted—maybe it's deadlocked or has exhausted file descriptors. The readiness endpoint gates whether the pod receives production traffic. During those critical first seconds after startup, while the service is establishing database connections, warming caches, or loading configuration from Secret Manager, readiness returns 503. The load balancer knows to wait.

In real production code, your readiness check would verify actual dependencies. Can you ping the database? Is Redis responding? Have you loaded your ML model into memory? For the billing service specifically, you might check whether Stripe SDK initialization completed or whether fraud detection rules loaded successfully. The randomness in the health check here simulates intermittent failures you'll encounter in production—network blips, transient resource exhaustion, or external dependency hiccups.

3.2 Containerization: Dockerfile Example

Once your service properly exposes its state, packaging it for cloud-native deployment becomes straightforward. I keep Dockerfiles deliberately simple and focused:

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY billing_service.py .
RUN pip install flask
EXPOSE 8080
CMD ["python", "billing_service.py"]

In production, you'd want to enhance this with multi-stage builds to minimize image size, run as a non-root user for security, and potentially use a requirements.txt for dependency management. But the core pattern remains: a slim base image, minimal layers, clear entrypoint. I've found that optimizing container startup time is one of the highest-leverage improvements you can make for intelligent load balancing, since faster startups mean less time in "not ready" state and smoother scaling.

3.3 GCP Resource Provisioning: Building and Deploying

With your service containerized, the next step is getting it into GCP's artifact registry and onto your cluster. I typically structure this as a repeatable pipeline, but here's the manual workflow to understand what's happening under the hood:

# Build, tag, and push Docker image to GCP Artifact Registry
gcloud artifacts repositories create python-services --repository-format=docker --location=us-central1

docker build -t us-central1-docker.pkg.dev/${PROJECT_ID}/python-services/billing-service:v1 .

gcloud auth configure-docker us-central1-docker.pkg.dev

docker push us-central1-docker.pkg.dev/${PROJECT_ID}/python-services/billing-service:v1

What matters here is that you're using Artifact Registry rather than Container Registry. Artifact Registry gives you vulnerability scanning out of the box, better IAM integration, and regional replication options that become critical when you're running multi-region services. I've migrated several production systems from Container Registry to Artifact Registry, and the improved security posture alone justified the effort.

Now comes the deployment configuration, which is where intelligent load balancing really starts to take shape:

# k8s/billing-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: billing-service
  template:
    metadata:
      labels:
        app: billing-service
    spec:
      containers:
      - name: billing-service
        image: us-central1-docker.pkg.dev/YOUR_PROJECT/python-services/billing-service:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Notice the probe configuration. I'm checking health every 5 seconds, which in production might be too aggressive depending on your service characteristics. You'll need to tune these values based on actual behavior. If health checks themselves become a source of load, lengthen the period. If you need faster failure detection, shorten it—but be prepared for more false positives during transient issues.

The initialDelaySeconds setting is critical and often misconfigured. Set it too short, and your pods fail health checks during normal startup, creating a restart loop. Set it too long, and you waste time before traffic can flow to newly scaled pods. I typically start with a value 2x my observed startup time in development, then tune based on production metrics.

Deploy the service and expose it with these commands:

kubectl apply -f k8s/billing-deployment.yaml
kubectl expose deployment billing-service --type=LoadBalancer --port 80 --target-port 8080

This creates a GCP Load Balancer in front of your deployment automatically, which brings us to the next layer of intelligence.

3.4 GCP Load Balancer with Intelligent Health Checks

When you create a LoadBalancer-type Kubernetes Service, GCP provisions an HTTP(S) Load Balancer that integrates deeply with your GKE cluster. This isn't just forwarding traffic—it's actively monitoring backend health, respecting readiness states, and making routing decisions millisecond by millisecond.

The real power comes from enabling container-native load balancing through Network Endpoint Groups (NEGs). This allows the GCP load balancer to route directly to pod IPs rather than going through kube-proxy and iptables, reducing latency and improving health check accuracy:

# k8s/billing-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: billing-service
  annotations:
    cloud.google.com/neg: '{"ingress": true}' # Enables container-native load balancing
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: billing-service

That single annotation—cloud.google.com/neg—transforms your load balancing architecture. I've measured 20-30% latency improvements in production just from enabling NEGs, because you're eliminating a network hop and iptables processing. More importantly for our purposes, it gives the GCP load balancer direct visibility into pod health. When a readiness probe fails, that backend is instantly removed from the load balancer's rotation. No eventual consistency, no delay waiting for endpoints to update.

Once deployed, you can fine-tune health check behavior through the GCP Console or gcloud commands. In production, I typically adjust the health check interval to balance between rapid failure detection and overhead. I also configure the unhealthy threshold—how many consecutive failures before removing a backend—based on whether I prefer availability (tolerate transient failures) or reliability (fail fast). For a billing service handling payments, I lean toward aggressive failure detection since partial failures can mean dropped transactions.

4. Deploying for Readiness, Scaling, and Resilience

4.1 Enabling Horizontal Pod Autoscaling

Intelligent load balancing doesn't just mean routing effectively to existing backends—it means ensuring you have the right number of healthy backends available at all times. This is where Kubernetes' Horizontal Pod Autoscaler becomes essential, working in concert with your load balancing strategy.

The beauty of combining proper health checks with autoscaling is that new pods only enter the load balancer rotation once they're actually ready. There's no race condition where traffic hits a pod that's still initializing. Here's how I typically configure autoscaling for a service like billing:

kubectl autoscale deployment billing-service --cpu-percent=70 --min=3 --max=10

I've learned through painful experience that setting the minimum replica count is just as important as the maximum. Running with fewer than 3 replicas in production means any single pod failure or deployment represents a significant percentage of your capacity, leading to cascading overload. With 3 minimum replicas across multiple availability zones, you maintain headroom even during disruptions.

The CPU threshold of 70% is conservative, which I prefer for services handling financial transactions. For less critical services, you might push to 80-85% to maximize resource efficiency. But here's what matters: combining autoscaling with readiness probes means traffic surges are handled gracefully. New pods spin up, initialize properly (blocked from traffic by readiness), then seamlessly join the load balancer pool once prepared.

In more sophisticated setups, I've extended this to use custom metrics—scaling based on request queue depth or P95 latency rather than just CPU. GCP makes this possible through the Custom Metrics API, allowing your application to export business-logic-aware metrics that drive scaling decisions. For a billing service, you might scale based on pending payment jobs rather than generic CPU usage.

4.2 Fine-Grained Traffic Splitting for Safe Deployments

Even with intelligent health checks and autoscaling, deploying new code remains the highest-risk operation in production. A bug that makes it past staging can take down your entire service if rolled out to all pods simultaneously. This is where traffic splitting and canary deployments become crucial, and where GKE's integration with GCP load balancing really shines.

The pattern I use most frequently is a canary deployment with percentage-based traffic splitting. You deploy a new version to a small number of pods while maintaining your stable version, then gradually shift traffic based on observed health metrics. Here's a canary deployment configuration:

# k8s/billing-deployment-canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: billing-service
      version: canary
  template:
    metadata:
      labels:
        app: billing-service
        version: canary
    spec:
      containers:
      - name: billing-service
        image: us-central1-docker.pkg.dev/YOUR_PROJECT/python-services/billing-service:v2
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Your Service selector includes both stable and canary version labels, so traffic flows to both. Initially with just 1 canary replica versus 3 stable replicas, roughly 25% of traffic hits the new version. You monitor error rates, latency, and business metrics. If everything looks healthy after an hour, you increase canary replicas to 2, then 3, then eventually promote it to stable while decommissioning the old version.

What makes this powerful is how it interacts with health checks. If your canary version has a critical bug that causes it to fail readiness probes, it never receives production traffic in the first place. The deployment completes, the pod starts, but the load balancer keeps routing around it. You discover the issue through monitoring rather than customer impact.

For even more sophisticated deployments, GCP's Traffic Director enables precise traffic splitting percentages, header-based routing for testing specific scenarios, and integration with service mesh capabilities. In one production system I worked on, we routed internal employee traffic to canary versions while keeping all customer traffic on stable, giving us real-world testing without customer risk.

5. Observability: Monitoring Health, Latency, and Failures

5.1 Logging and Monitoring with Cloud Operations Suite

Here's the uncomfortable truth about load balancing: you can architect the most sophisticated routing logic in the world, but without observability, you're blind to whether it's actually working. Intelligent load balancing requires data—continuous, detailed data about pod health, request latency, error rates, and traffic distribution.

This is where GCP's Cloud Operations Suite becomes indispensable. The integration with GKE is deep enough that you get pod-level metrics, container logs, and distributed traces with minimal configuration. But getting the most value requires instrumenting your services to export meaningful data that can drive routing decisions.

For the billing service, I export several classes of metrics. First, the basics—request count, error rate, latency percentiles. These flow automatically through GCP's managed Prometheus if you expose them in the right format. Second, health check results over time, which helps identify patterns in failures. Is a pod failing health checks every morning at 2am during database maintenance? That's a signal to tune your health check logic or adjust maintenance windows.

Third, and most importantly, custom business metrics that represent actual service health from a user perspective. For billing, that might be payment success rate, time to process refunds, or fraud detection latency. These metrics inform autoscaling, alerting, and ultimately load balancing decisions.

Here's how to export custom metrics using OpenTelemetry from your Flask service:

# Export Flask metrics (latency, errors) using OpenTelemetry
from opentelemetry import metrics
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

exporter = CloudMonitoringMetricsExporter()
meter_provider = MeterProvider(
    metric_readers=[PeriodicExportingMetricReader(exporter, export_interval_millis=5000)]
)
metrics.set_meter_provider(meter_provider)

meter = metrics.get_meter(__name__)
payment_latency = meter.create_histogram(
    "billing.payment.latency",
    unit="ms",
    description="Payment processing latency"
)

# In your endpoint:
@app.route("/pay", methods=["POST"])
def pay():
    start = time.time()
    # ... process payment ...
    duration_ms = (time.time() - start) * 1000
    payment_latency.record(duration_ms)
    return jsonify({"status": "success"})

With these metrics flowing to Cloud Monitoring, your SRE team can make informed decisions. When should you scale? When is a canary actually safer than the stable version? Which pods are consistently slower than their peers? I've built dashboards that show per-pod latency distributions, making it immediately obvious when a single pod is degraded. That visibility has prevented countless incidents by enabling preemptive action before customers notice problems.

The other critical piece is tracing. Cloud Trace integration with GKE means you can follow a request from the load balancer through your billing service and into downstream calls to payment processors. When P95 latency spikes, you can pinpoint whether it's your code, database queries, or external API calls. This depth of visibility transforms troubleshooting from guesswork into data-driven investigation.

5.2 Alerting on Failures and Degrading Latency

Observability data is useless unless it drives action. I configure alert policies that treat different signal types appropriately—some require immediate pages, others just create tickets for investigation during business hours.

For the billing service, critical alerts include error rate exceeding 1% sustained over 5 minutes, or any instance of payment processing failing for all attempts in a 2-minute window. These page whoever is on-call because they represent immediate customer impact. Medium-severity alerts might fire when P95 latency exceeds 1 second, or when a pod fails health checks more than 3 times in 10 minutes. These create tickets but don't page—they indicate degraded performance that needs investigation but isn't yet critical.

The key is connecting alerts to automated responses where possible. When error rate spikes on canary pods, automatically roll back the deployment. When autoscaling maxes out capacity, notify the on-call engineer to investigate whether you need to increase limits or optimize performance. When a pod consistently fails health checks after startup, kill it and let Kubernetes reschedule—maybe it landed on a degraded node.

I've built automation around these alerts using Cloud Functions triggered by Pub/Sub messages from Cloud Monitoring. The function can scale deployments, restart pods, or even drain traffic from an entire cluster if metrics indicate a zone-level failure. This closes the loop from observation to intelligent action without requiring human intervention for common scenarios.

6. Secure Networking, IAM, and Service Access

6.1 Restricting Internal Traffic with VPCs

Intelligent load balancing isn't just about routing efficiency—it's also about security. Production SaaS systems need defense in depth, where compromising one service doesn't grant access to your entire infrastructure. This is where network policies and VPC configuration become part of your load balancing strategy.

I deploy production GKE clusters as private clusters, meaning nodes don't have public IP addresses and can't be reached from the internet except through the load balancer. Within the cluster, I use Kubernetes NetworkPolicies to enforce which services can communicate:

# k8s/network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: billing-allow-internal
spec:
  podSelector:
    matchLabels:
      app: billing-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway

This policy ensures that only pods labeled app: api-gateway can initiate connections to billing service pods. If an attacker compromises your notification service, they can't directly access billing. They'd need to pivot through the gateway, which is more heavily monitored and locked down.

I've seen incidents where network policies prevented lateral movement after a container escape vulnerability. The attacker got pod access but couldn't reach any valuable services because network policies blocked the traffic. It bought enough time to detect and respond before data was compromised.

The policies also interact with intelligent load balancing in subtle ways. By restricting which services can reach your backends, you ensure all external traffic flows through the load balancer where it's subject to health checks, rate limiting, and observability. Internal service-to-service calls might bypass the load balancer for efficiency, but they're still subject to network policies and service mesh controls if you're running Istio or similar.

6.2 IAM Controls: Least Privilege

Network policies handle network-level access, but IAM controls what authenticated services can do. I configure every microservice with its own Kubernetes Service Account mapped to a specific GCP Service Account through Workload Identity. The billing service needs access to Cloud SQL for transaction records and Pub/Sub for publishing payment events, but nothing else.

This principle of least privilege has saved me multiple times. In one incident, a vulnerability in a dependency allowed arbitrary code execution in the notification service. Because that service's IAM permissions were tightly scoped to only send emails via SendGrid, the attacker couldn't access customer payment data, couldn't modify infrastructure, couldn't even list what other services existed. The blast radius was contained to what we could tolerate.

When combined with intelligent load balancing and health checks, IAM controls ensure that even if a compromised pod passes health checks and receives traffic, the damage it can do is minimized. You've created a system that degrades gracefully even under active attack, continuing to serve legitimate users while containing the compromise.

7. Production Scenario: Handling a Real Failure

Theory is satisfying, but what matters is how this architecture performs when things go wrong. Here's a scenario I've lived through, with names changed: You deploy a new version of billing-service v2.1.4 that includes an optimization for batch processing. The change looks good in staging. You roll it out as a canary to 10% of production traffic.

Within minutes, P95 latency for requests hitting the canary pod jumps from 200ms to 3 seconds. Error rate climbs from 0.1% to 2%. In the old architecture, this would mean 10% of your users are having a terrible experience, and you'd be racing to roll back manually while your support team fields angry tickets.

Instead, here's what happens with intelligent load balancing: The canary pod's readiness probe starts failing because you've configured it to check not just "is the process alive" but "are recent requests completing successfully." After 3 consecutive failures, Kubernetes marks the pod as not ready. The GCP load balancer immediately stops routing new traffic to it, even though the pod is still running. Your healthy stable pods absorb the additional load, and autoscaling spins up an extra stable pod to handle the increased traffic.

Cloud Monitoring detects the pattern—canary pods failing health checks, latency spike isolated to v2.1.4. An alert fires to your Slack channel. Your automated rollback policy kicks in because the canary exceeded failure thresholds. Within 2 minutes of the initial deployment, the canary is removed, and you're back to running entirely on stable v2.1.3. Total customer impact: a few dozen requests saw elevated latency before the health check failed. No one noticed.

Your on-call engineer investigates the next morning rather than at 2am. Looking at traces in Cloud Trace, they discover the optimization introduced a database query that locks tables during batch operations, blocking interactive requests. It's fixed in v2.1.5, which passes canary validation and rolls out smoothly.

This is the promise of intelligent load balancing—not that systems never fail, but that they fail gracefully, contain the blast radius, and provide the visibility needed to fix problems without drama.

8. Common Pitfalls and Best Practices

Even with the architecture I've described, there are failure modes I've encountered that are worth calling out explicitly. The most common mistake I see is teams implementing health and readiness probes that check the wrong things. Your probe might verify that Flask is responding, but not whether the database connection pool is exhausted. It might return 200 OK while background threads are deadlocked. Effective probes check whether the service can actually fulfill its purpose, not just whether the process is running.

Another pitfall is tuning health check intervals without considering the full impact. Very aggressive checking (every second) can overwhelm your application with probe traffic, especially if the health check itself is expensive. But very conservative checking (every 30 seconds) means it can take over a minute to detect a failed pod and remove it from rotation. I've found that 5-10 second intervals strike a good balance for most services, but you need to measure in your own environment.

The fail-open versus fail-closed decision is subtle but critical. When your load balancer has multiple unhealthy backends, should it continue routing to them (fail-open) or refuse traffic entirely (fail-closed)? The right answer depends on your service. For a billing system, I prefer fail-closed—better to return 503 and have clients retry than to process payments incorrectly. For a recommendation engine, fail-open might be better—showing slightly stale recommendations is preferable to showing nothing.

I always advocate for testing failure scenarios in production with tools like chaos engineering. Use kubectl delete pod to verify that traffic smoothly fails over to healthy pods. Use network policies to simulate latency or packet loss. Inject failures in your canary deployments intentionally to verify monitoring catches them. Every production service I run has regular chaos experiments scheduled because the confidence they provide is invaluable.

Finally, load testing is non-negotiable. Use tools like Locust or k6 to simulate realistic traffic patterns and verify that autoscaling responds appropriately, that health checks remain reliable under load, and that your performance assumptions hold. I've discovered countless issues during load tests that never manifested in staging with synthetic traffic.

9. Conclusions and Final Thoughts

The modern SaaS backend is both a distributed system and a living organism—adapting, self-healing, and scaling on demand. What I've described in this article isn't just theoretical architecture; it's the pattern I've refined across dozens of production systems, validated through incidents that ranged from minor hiccups to company-threatening outages.

The real insight, which took me years to internalize, is that intelligent load balancing isn't a feature you add at the end. It's an emergent property of good architecture: services that honestly report their state, infrastructure that respects those signals, and observability that closes the feedback loop. When these pieces align, you get a system that routes traffic not based on naive heuristics, but on genuine understanding of backend health and capacity.

GCP's managed services make this accessible in ways that weren't possible a decade ago. The deep integration between GKE, Cloud Load Balancing, and Cloud Operations means you're not duct-taping together disparate tools—you're working with a coherent platform where health checks flow naturally into routing decisions, where metrics inform autoscaling, and where the blast radius of failures is naturally contained.

But the technology is only half the story. The teams that succeed with architectures like this are those who obsessively observe their systems in production, who treat every incident as a learning opportunity, and who iterate relentlessly on their traffic control strategies. The advice I've shared comes not from planning but from responding—to cascading failures at 3am, to traffic spikes during product launches, to subtle bugs that only manifest at scale.

If you take one thing from this article, let it be this: intelligent load balancing is about building systems that fail gracefully and heal automatically, giving you the breathing room to fix problems thoughtfully rather than frantically. It's about creating that invisible engine—fast, resilient, secure, and ready for any growth you throw at it. And perhaps most importantly, it's about letting you sleep through the night while your infrastructure handles the inevitable chaos of production without human intervention.

The patterns I've shared are battle-tested, but they're not prescriptive. Your SaaS will have different constraints, different failure modes, different business requirements. Adapt these concepts to your context, measure what matters for your services, and build the observability that lets you iterate with confidence. That's how you evolve from naive load balancing to truly intelligent traffic management—one production incident at a time.

Authenticating REST services with OAuth2

Juan Carlos González Cabrero — Fri, 06 Aug 2021 22:20:07 +0000

1. Introduction

When it comes to adding authorization to call secured services, we realize not only that the configuration changes depending on which framework you are going to use, but that for each HTTP client you use, you must configure OAuth2 in a different way.

For this reason, the simplest thing when implementing an authorization layer through OAuth2 to call those services, would be to outsource the generation of the tokens to a new personalized client. This way we would have a maintainable integration, isolated from the REST client we are using.

This article guides you through the creation of a simple library which allows you to grant your HTTP requests with the required authorization token, and integrate in your services whatever client you may use.

The authorization flow is described in the image above:

Authorization request is sent from client to OAuth server.
Access token is returned to the client.
Access token is then sent from client to the API service (acting as resource server) on each request for a protected resource access.
Resource server checks the token with the OAuth server, to confirm the client is authorized to consume that resource.
Server responds with requested protected resources.

2. Setting up the required dependencies

We will need a few libraries to build our custom OAuth2 client.

First of all, the Apache HTTP client library, which will provide us with the HTTP client for the integration with the authorization server, as well as a toolset for the request building. So it would be the core library for our client.

In the second one, we find another Apache library, called cxf-rt-rs-security-oauth2. In this case, this dependency would be optional, since we only need a set of predefined values in the OAuth2 Protocol definition, gathered in the OAuthConstants class. We could also define those values by ourselves, to get rid of this dependency.

Lastly, we include the json library. This library is a helpful toolset when we are handling JSON data. It is really useful to parse and manipulate JSON in Java.

<dependencies>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5</version>
    </dependency>

    <dependency>
        <groupId>org.apache.cxf</groupId>
        <artifactId>cxf-rt-rs-security-oauth2</artifactId>
        <version>3.4.2</version>
    </dependency>

    <dependency>
        <groupId>org.json</groupId>
        <artifactId>json</artifactId>
        <version>20160212</version>
    </dependency>
</dependencies>

3. Building the OAuth2 request

We have to build the request to the server which will authorize our service as a granted client.
To achieve this, we need to define the OAuth2 configuration we are using, including the grant type, the authorization server URL, the credentials for the given grant type, and the scope for the resource we are requesting.

class OAuth2Config {
  String grantType;
  String clientId;
  String clientSecret;
  String username;
  String password;
  String accessTokenUri;
  String scope;
}

Once we have the configuration values initialized, we can use them to build the HTTP request for the authorization server.
Typically, the HTTP method used to get the access token, will be a POST, as defined in the OAuth 2.0 Authorization Protocol specification:

The client MUST use the HTTP "POST" method when making access token requests.

Depending on the grant type we define, we must define different parameters on the POST request. We will use a list of NameValuePair to gather all those needed parameters.

HttpUriRequest buildRequest() {
  List<NameValuePair> formData = new ArrayList<>();
  formData.add(new BasicNameValuePair(GRANT_TYPE, config.getGrantType()));

  if (config.getScope() != null && !config.getScope().isBlank()) {
    formData.add(new BasicNameValuePair(SCOPE, config.getScope()));
  }

  if (CLIENT_CREDENTIALS_GRANT.equals(config.getGrantType())) {
    formData.add(new BasicNameValuePair(CLIENT_ID, config.getClientId()));
    formData.add(new BasicNameValuePair(CLIENT_SECRET, config.getClientSecret()));
  }

  if (RESOURCE_OWNER_GRANT.equals(config.getGrantType())) {
    formData.add(new BasicNameValuePair(RESOURCE_OWNER_NAME, config.getUsername()));
    formData.add(new BasicNameValuePair(RESOURCE_OWNER_PASSWORD, config.getPassword()));
  }

  return RequestBuilder.create(HttpPost.METHOD_NAME)
                       .setUri(config.getAccessTokenUri())
                       .setEntity(new UrlEncodedFormEntity(formData, StandardCharsets.UTF_8))
                       .build();
}

4. Executing the OAuth2 request

Since we are building an OAuth2 client as basic as possible, we will use the default HTTP client from Apache HTTP library, to send our request to the authorization server.

CloseableHttpResponse doRequest(HttpUriRequest request) {
  CloseableHttpClient httpClient = HttpClients.createDefault();
  try {
    return httpClient.execute(request);
  } catch (IOException e) {
    throw new OAuth2ClientException("An error occurred executing the request.", e);
  }
}

Once we receive the response, we need to handle it, extracting the information we need for the access token.

class OAuth2Response {
  HttpEntity httpEntity;
}

We should check for errors before parsing the content to get the access token. We can consider here errors in the credentials we defined, a wrong or malformed URL, or any internal error from the authorization server.

OAuth2Response execute(HttpUriRequest request) {
  CloseableHttpResponse httpResponse = doRequest(request);
  HttpEntity httpEntity              = httpResponse.getEntity();
  int statusCode                     = httpResponse.getStatusLine()
                                                   .getStatusCode();
  if (statusCode >= 400) {
    throw new OAuth2ClientException(statusCode, httpEntity);
  }
  return new OAuth2Response(httpEntity);
}

We should not forget to close the httpResponse, to avoid the memory leakage. But it is pretty important to wait until it is read properly, since it contains an InputStream which would become inaccessible once we have closed it.

Typically, the response content will come on a JSON format, with the access token data in a key-value schema. However, we should consider a server handling the data in a different format, like XML or URL encoded.

For the scope of this article, we will consider our authorization server giving us JSON formatted content. The org.json:json library we included earlier, will help us on the deserialization.

JSONObject handleResponse(HttpEntity entity) {
  String content     = extractEntityContent(entity);
  String contentType = Optional.ofNullable(entity.getContentType())
                               .map(Header::getValue)
                               .orElse(APPLICATION_JSON.getMimeType());
  return new JSONObject(content);
}

String extractEntityContent(HttpEntity entity) {
  try {
    return EntityUtils.toString(entity, StandardCharsets.UTF_8);
  } catch (IOException e) {
    throw new OAuth2ClientException("An error occurred while extracting entity content.", e);
  }
}

Given the JSONObject, it becomes much easier to handle the response, since we can retrieve instantly each value we are interested in.

5. Putting all together

The goal here is to obtain an access token to call the secured services we need. However, sometimes we also need to know some additional data, like the timestamp when the token is going to expire, the token type we are receiving, or the refresh token in the case the grant type is defined so.

class AccessToken {
  long expiresIn;
  String tokenType;
  String refreshToken;
  String accessToken;

  AccessToken(JSONObject jsonObject) {
    expiresIn    = jsonObject.optLong(ACCESS_TOKEN_EXPIRES_IN);
    tokenType    = jsonObject.optString(ACCESS_TOKEN_TYPE);
    refreshToken = jsonObject.optString(REFRESH_TOKEN);
    accessToken  = jsonObject.optString(ACCESS_TOKEN);
  }
}

Finally, we will get a client which will retrieve the access token data needed to grant our calls to the services, based on the configuration we defined.

AccessToken accessToken() {
  HttpUriRequest request  = buildRequest();
  OAuth2Response response = execute(request);

  return new AccessToken(handleResponse(response.getHttpEntity()));
}

6. Put into practice

But, how could we integrate this custom client in our service?

Well, as I mentioned at the beginning of the article, the idea of this custom OAuth2 client is to be isolated from the framework and/or the HTTP client we are using to consume the secured services.

So I will show you a few examples of how to integrate it in different service environments.

6.1. Spring Framework - WebClient

class WebClientConfig {

  @Bean(name = "securedWebClient")
  WebClient fetchWebClient(@Value("${host}") String host,
                           OAuth2Config oAuth2Config) {
    OAuth2Client oAuth2Client = OAuth2Client.withConfig(oAuth2Config).build();
    return WebClient.builder()
                    .filter(new OAuth2ExchangeFilter(oAuth2Client))
                    .baseUrl(host)
                    .build();
  }

  @Bean
  @ConfigurationProperties(prefix = "security.oauth2.config")
  OAuth2Config oAuth2Config() {
    return new OAuth2Config();
  }

  class OAuth2ExchangeFilter implements ExchangeFilterFunction {

    OAuth2Client oAuth2Client;

    @Override
    public Mono<ClientResponse> filter(ClientRequest request,
                                       ExchangeFunction next) {
      String token = Optional.ofNullable(oAuth2Client.accessToken())
                             .map(AccessToken::getAccessToken)
                             .map("Bearer "::concat)
                             .orElseThrow(() -> new AccessDeniedException());

      ClientRequest newRequest = ClientRequest.from(request)
                                              .header(HttpHeaders.AUTHORIZATION, token)
                                              .build();
      return next.exchange(newRequest);
    }
  }
}

6.2. Spring Framework - Feign Client

class FeignClientConfig {

  @Bean
  OAuthRequestInterceptor repositoryClientOAuth2Interceptor(OAuth2Client oAuth2Client) {
    return new OAuthRequestInterceptor(oAuth2Client);
  }

  class OAuthRequestInterceptor implements RequestInterceptor {

    OAuth2Client oAuth2Client;

    @Override
    public void apply(RequestTemplate requestTemplate) {
      String authToken = Optional.ofNullable(oAuth2Client.accessToken())
                                 .map(AccessToken::getAccessToken)
                                 .map("Bearer "::concat)
                                 .orElseThrow(() -> new AccessDeniedException());

      requestTemplate.header(HttpHeaders.AUTHORIZATION, authToken);
    }
  }
}

6.3. Vert.x - Web Client

class ProtectedResourceHandler implements Handler<RoutingContext>  {

  OAuth2Config oAuth2Config;

  ProtectedResourceHandler() {
    // Resource handler initialization
    oAuth2Config = oauth2Config(config);
  }

  private OAuth2Config oauth2Config(JsonObject oauth2Properties) {

    return OAuth2Config.builder()
        .grantType(oauth2Properties.getString("grantType"))
        .accessTokenUri(oauth2Properties.getString("accessTokenUri"))
        .clientId(oauth2Properties.getString("clientId"))
        .clientSecret(oauth2Properties.getString("clientSecret"))
        .username(oauth2Properties.getString("username"))
        .password(oauth2Properties.getString("password"))
        .scope(oauth2Properties.getString("scope"))
        .build();
  }

  @Override
  public void handle(RoutingContext routingContext) {

    WebClient.create(routingContext.vertx())
        .getAbs(host)
        .uri(endpoint)
        .putHeader(HttpHeaders.AUTHORIZATION.toString(), generateToken())
        .send()
        .onSuccess(httpResponse -> { /* Successful response handler */ })
        .onFailure(err -> { /* Error response handler */ });
  }

  String generateToken() {

    return Optional.of(OAuth2Client.withConfig(oAuth2Config).build())
        .map(OAuth2Client::accessToken)
        .map(AccessToken::getAccessToken)
        .map("Bearer "::concat)
        .orElseThrow(() -> new AccessDeniedException());
  }
}

6.4. Quarkus - RESTEasy

@RegisterRestClient
@RegisterClientHeaders(SecurityHeaderFactory.class)
interface DocumentClient {

  // External endpoints definition
}

class SecurityHeaderFactory implements ClientHeadersFactory {

  OAuth2Client oAuth2Client;

  @Inject
  SecurityHeaderFactory(OAuth2Properties oAuth2Properties) {
    oAuth2Client = OAuth2Client
        .withConfig(oauth2Config(oAuth2Properties))
        .build();
  }

  @Override
  public MultivaluedMap<String, String> update(MultivaluedMap<String, String> incomingHeaders,
                                               MultivaluedMap<String, String> outgoingHeaders) {
    outgoingHeaders.add(HttpHeaders.AUTHORIZATION.toString(), generateToken());
    return outgoingHeaders;
  }

  String generateToken() {
    return Optional.of(oAuth2Client)
        .map(OAuth2Client::accessToken)
        .map(AccessToken::getAccessToken)
        .map("Bearer "::concat)
        .orElseThrow(() -> new AccessDeniedException());
  }

  OAuth2Config oauth2Config(OAuth2Properties oAuth2Properties) {
    return OAuth2Config.builder()
        .grantType(oAuth2Properties.getGrantType())
        .accessTokenUri(oAuth2Properties.getAccessTokenUri())
        .clientId(oAuth2Properties.getClientId())
        .clientSecret(oAuth2Properties.getClientSecret())
        .username(oAuth2Properties.getUsername())
        .password(oAuth2Properties.getPassword())
        .scope(oAuth2Properties.getScope())
        .build();
  }
}

7. Conclusion

In this article, we have seen how we can set up a simple OAuth2 Client, and how we can integrate it in your REST calls to retrieve a secured resource from an external service.

You can check the code used for the OAuth2 Client, the repository is available over on Github.