Forem: Rafał Szczepanik

Docker Compose and Devcontainers for Microservices Development

Rafał Szczepanik — Mon, 28 Apr 2025 16:12:40 +0000

Introduction

Imagine a scenario - you are working as a developer in a fast-growing company. On board there are many teams, developing separate services that intertwine between each other. Let's say you have:

App 1 - developed mostly by Team A, with occasional changes by Team B and C. It communicates with App 2 and App 3 via REST.
App 2 - developed primarily by Team B and A. It serves as an API for both App 1 and 3, while also communicating with other microservices via GRPC or Kafka.
App 3 - developed mostly by Team C, with support from Team A, and occasional changes by Team B. This one interacts with microservice Y.
…plus 5 more microservices, each with its own dependencies, databases, or message queues, developed by various combinations of teams.

The point is - developers frequently need to work on multiple services, often simultaneously. They need to see how their changes affect service they are working on, and all dependent services - all without the friction and delay of deploying to a shared staging environment. Onboarding new team members needs to be swift, not a week-long setup marathon wrestling with dependencies.

So how to approach this problem?

Traditional local development

The approach we all know. In this flow, we setup every service we need locally on our machine. Manually installing all dependencies, following steps in README. This is good enough for single repositories, however will come with problems at scale:

Port management - every service has to have its own set of ports. These addresses will have to be configured in other services. It is not easy to tell, what address localhost:3000 , localhost:3001 or localhost:3010 is responsible for.
Dependency management - every project will need their own versions of NodeJS, Golang, Ruby, etc…
"Works on my machine" - differences in OS, installed packages, and configurations leading to inconsistencies.
CORS, Webstorage API, etc. - different ports mean different origins. This causes many issues for frontend apps, like blocked requests or missing cookies.
Onboarding difficulties - every new team member has to go through many README files. Every one of this readme's will differ from each other.

Devcontainers with Docker Compose and Traefik

This tutorial assumes familiarity with Docker, Docker Compose, Devcontainers and that your services have Dockerfile implemented

Concept Overview

The solution consists of:

A central repository containing:
- A makefile with utilities (e.g., init to pull service repositories)
- A docker-compose configuration for orchestrating services
Service repositories with:
- Dockerfiles for containerization
- Devcontainer configurations for development
Traefik integration:
- Routes traffic to services. For example, you can have multiple frontends accessible on the same domain, like localhost:3000/svc1 , localhost:3000/svc2.
- Provides a dashboard (localhost:5555 by default) to inspect configurations of routings.

Source: Traefik Docker Provider Documentation

Key Benefits

Flexible service management:
- Run selected services using COMPOSE_PROFILES
- Stop, rebuild, and restart services with ease
Seamless development experience:
- Switch between running containerized services and developing in devcontainers
- Maintain identical networking configurations in both environments
- No more conflicting ports

Setup base compose repository

The first step is to create an empty repository, then inside it create a Makefile with essential procedures:

REPOS = \
    git@github.com:some/repository1.git \
    git@github.com:some/repository2.git \
    git@github.com:some/repository3.git \
    ...

.PHONY: init dev down

$(REPO_DIRS):
    git clone --filter=blob:none $(filter %$@.git,$(REPOS));

# Responsible for pulling repositories
init: $(REPO_DIRS)

# Will be used as a shortcut to start all services
dev:
    docker compose up -d

# Stops and removes all docker services
down:
    COMPOSE_PROFILES=all docker compose down

For Traefik to work correctly, we first need to add a traefik.yml configuration file:

log:
  level: INFO

api:
  dashboard: true
  insecure: true

providers:
  docker:
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: false
    # This must match the network name used by your services
    network: shared-network

ping: {}

Next, let's create compose.yml. In this file we will:

Create a Traefik service to handle routing between our microservices
Define all our microservices with appropriate configurations
Create an attachable network, allowing devcontainers to connect to the same network
Assign profiles to each service for selective launching:
- Each service gets its own profile (same as the service name)
- All services get the all profile for stopping all services at once
- You can group related services by assigning the same profile name to both. For example, if a database is useless without its API service, give both the same profile name so they always start together
Configure the build process for each service, including environment variable handling

Here's the complete compose.yml configuration:

services:
  traefik:
    image: traefik:v2.11.0
    ports:
      - 3000:80 # Services you expose will be visible on localhost:3000/**
      - 5555:8080 # This is a port for a traefik dashboard
    volumes:
      - ./traefik.yml:/etc/traefik/traefik.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: --configFile=/etc/traefik/traefik.yml
    networks:
      - default
    postgres-db:
    profiles:
      - postgres-db
      - all
    image: postgres:16.2
    networks:
        - default
    ...

  # Lets assume svc1 is one of your repositories
    svc1:
        hostname: svc1
        profiles:
        - svc1
        - all
        image: svc1:latest
        env_file:
        # here is a list of env files, that will be loaded
        # when service is started
        # (important - these won't be loaded for build process, for that
        # we will use something else)
      - svc1/.env.example
      - path: svc2/.env.local # if exists, .env.local will override variables coming from .env.example
        required: false
        networks:
            - default
        build:
            context: ./svc1
            dockerfile: Dockerfile
            args:
            # these arguments will be passed to Dockerfile ARGS
            # you can't use here directly env files, so you have to
            # set them manually to some env variable, along with default value:
                - API_KEY=${API_KEY:-your_default_api_key}
                - LOG_LEVEL=${LOG_LEVEL:-info}

        labels:
            # If you want to expose this service on localhost:3000/svc1 add this
            # Will route any traffic matching the rule to svc1 on port 5173
      - traefik.enable=true
      - traefik.http.services.svc1.loadbalancer.server.port=5173
      - traefik.http.routers.svc1.rule=(Host(`localhost`) || Host(`host.docker.internal`)) && PathPrefix(`/svc1`)

        ...

  # Placeholder for another service (e.g., svc2)
  svc2:
    hostname: svc2
    profiles:
    - svc2
    - all
    image: svc2:latest
    env_file:
      - svc2/.env.example
    networks:
      - default
    build:
      context: ./svc2
      dockerfile: Dockerfile
    labels:
      # Expose svc2 on localhost:3000/svc2, routing to its internal port (e.g., 8080)
      - traefik.enable=true
      - traefik.http.services.svc2.loadbalancer.server.port=8080 # Adjust port as needed
      - traefik.http.routers.svc2.rule=(Host(`localhost`) || Host(`host.docker.internal`)) && PathPrefix(`/svc2`)

networks:
  default:
      # This is crucial for devcontainers to be able to connect to this network
    attachable: true
    name: shared-network

volumes:
    ...

Now let's create .env and .env.example files. The .env file is loaded automatically when running docker compose:

Example .env file:

COMPOSE_PROFILES=postgres-db,svc1,...

Now you can run make dev, which will start all services listed in the compose profiles. When an image is not available (like in svc1's case), it will build the image based on the instructions provided.

The build process can be tricky. The reason we added default values is that there's no easy way to access env files during the first launch.

Enhance rebuilding flow

Our setup revolves around keeping all environment files for services in their respective directories. Now let's implement a better process for rebuilding images.

We'll introduce a new build command in our Makefile, which will first load env files related to the service, and then rebuild the image:

To do this, add in your Makefile:

[... existing code ...]

# List of all env files, in order of least relevant to most relevant
# build procedure will try to load all of them. Customize for your use case.
ENV_FILES = .env.example .env.development .env .env.local .env.development.local

build:
    # If 'build' is called without specific service names, build all services defined in compose.
    @if [ -z "$(filter-out $@,$(MAKECMDGOALS))" ]; then \
        docker compose build; \
        exit 0; \
    fi
    # Get the list of service names passed as arguments (e.g., make build svc1 svc2)
    @services="$(filter-out $@,$(MAKECMDGOALS))"; \
    # Loop through each specified service name
    for service in $$services; do \
        echo "Building service: $$service"; \
        # Temporarily export variables sourced from env files so docker compose build can see them
        set -a; \
        # Loop through the predefined list of env file names/suffixes
        for env_suffix in $(ENV_FILES); do \
            # Check if the env file exists in the service's directory
            if [ -f "$$service/$$env_suffix" ]; then \
                # Source the file, loading its variables into the environment
                source "$$service/$$env_suffix"; \
            fi; \
        done; \
        # Stop exporting variables
        set +a; \
        docker compose build $$service || exit 1; \
    done

# needed because of passing arguments to make command
%:
    @:

Now when you run make build svc1, your service will be built with ARGS values defined in your repository's env files (or default ones if no env files are found).

Devcontainers in services

Currently, you have configured Docker Compose in a way that makes it easy to run and rebuild your services. They can communicate with each other because they're in the same network. But this setup is mainly useful when you're not actively developing these services.

To enable easy development, navigate to your service directory and create a .devcontainer directory. For simplicity, I'll present a configuration for a Node application.

Inside the directory, create two files:

devcontainer.json:

{
  "name": "Node",
  "dockerComposeFile": "./docker-compose.devcontainer.yaml",
  "service": "base",
  "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
  "customizations": {
    "vscode": {
      "settings": {
        "remote.autoForwardPorts": false
      },
      "extensions": [
          ... list of extensions, if any...
      ]
    }
  }
}

And docker-compose.devcontainer.yaml:

services:
  base:
    image: node:22.12.0
    command: sleep infinity
    hostname: svc1
    networks:
      - default
    volumes:
      - ../../:/workspaces:cached
    labels:
      # these has to be the same, as in main docker compose
      - traefik.enable=true
      - traefik.http.services.svc1.loadbalancer.server.port=5173
      - traefik.http.routers.svc1.rule=(Host(`localhost`) || Host(`host.docker.internal`)) && PathPrefix(`/svc1`)

networks:
  shared-network:
    external: true

And that's all! When you open this repository using a devcontainer, you'll have the whole environment ready for development. All routing to other services will work either using direct connection to other containers (for example http://svc2:3000), or using Traefik (http://host.docker.internal:3000/svc2). You will still have your local-development features, like hot-reload or debugging.

Because the devcontainer shares the shared-network with the services run by the main compose.yml, you can connect from your devcontainer (svc1) to other services (svc2) in two main ways:

Direct Service-to-Service: Use the service name defined in compose.yml as the hostname (e.g., fetch('http://svc2:8080/api/data')). This works because Docker's internal DNS resolves service names on the shared network. Use the internal port of the target service (e.g., 8080 for svc2 in our example).
Via Traefik: Access services through the Traefik proxy using the host machine's address and the path defined in the Traefik rule (e.g., fetch('http://host.docker.internal:3000/svc2/api/data')). host.docker.internal is a special DNS name that resolves to the host machine from within a Docker container. Traefik then routes the request based on the path (/svc2) to the correct container and port.

You will still have your local-development features, like hot-reload or debugging, within the devcontainer.

Summary

We've successfully created a development environment that allows teams to work efficiently on interconnected microservices without the usual friction:

Consistent environments across the team - no more "works on my machine" issues or week-long setup processes.
Flexible service management - choose which services to run based on your current task using profiles
Easy switching between modes - run services in containers when just consuming them, or switch to devcontainers when actively developing
Simplified networking - consistent service discovery regardless of whether you're running services or developing them

Use this guide as a baseline, and extend the setup according to your specific needs.

Scaling Next.js with Redis cache handler

Rafał Szczepanik — Mon, 10 Jun 2024 13:08:22 +0000

Let's say you have dozens of Next.js instances in production, running in your Kubernetes cluster. Most of your pages use Incremental Static Regeneration (ISR), allowing pages to be generated and saved in file storage upon a user's first visit. Subsequent requests to the same page are served instantly from the saved version, bypassing regeneration, at least until the set revalidation period expires. Sounds good, right?

Except it does not scale very well.

Problem

The data is generated but never cleaned up. Moreover, every instance of NextJS uses the same data, duplicated and isolated. Here at Odrabiamy.pl, we noticed that all of our k8s instances were taking up to 30GB of storage each. That is a massive amount of data for one node, but what if we have 20 nodes? That would be 600 GB of data, which could easily be shared.

Possible Solutions

We tried to come up with a solution to this problem, and these were our options:

Use a Kubernetes persistent volume and share the inside of the .next directory, but it has its cons:
1. Every pod would have read/write access, which could cause massive problems with race conditions between pods. We would have to write our own cache handler to make sure everything is stable.
2. A mechanism would be needed to copy the .next directory to a shared volume during deployment and, after it is not needed anymore, to delete it.
Use Redis and the existing Next.js config to store all the generated pages - which turned out to be perfect for us in terms of the required time to implement and the complexity of the solution.

Next.js and Redis

By default, Next.js uses a file-based cache handler. However, Vercel has published a new config option to customize that. To do this, we have to load a custom cache handler in our next.config.js:

  cacheHandler:
    process.env.NODE_ENV === 'production'
      ? require.resolve('./cache-handler.cjs')
      : undefined,

We only load it in the production environment, as it isn’t necessary in development mode. Now it is time to implement the cache-handler.cjs file. (Note: depending on your npm config, you might need to write this using ES modules.)

We will utilize the @neshca/cache-handler package, which is a library that comes with pre-written handlers. The plan is to:

Set Redis as the primary cache handler
As a backup, use LRU cache (Least Recently Used, in-memory cache)

The basic implementation will be as follows:

// cache-handler.cjs
const createClient = require('redis').createClient;

const CacheHandler = require('@neshca/cache-handler').CacheHandler;
const createLruCache = require('@neshca/cache-handler/local-lru').default;
const createRedisCache = require('@neshca/cache-handler/redis-strings').default;

CacheHandler.onCreation(async () => {
  const localCache = createLruCache({
    maxItemsNumber: 10000,
    maxItemSizeBytes: 1024 * 1024 * 250, // Limit to 250 MB
  });

  let redisCache;
  if (!process.env.REDIS_URL) {
    console.warn('REDIS_URL env is not set, using local cache only.');
  } else {
    try {
      const client = createClient({
        url: process.env.REDIS_URL,
      });

      client.on('error', (error) => {
        console.error('Redis error', error);
      });

      await client.connect();

      redisCache = createRedisCache({
        client,
        keyPrefix: `next-shared-cache-${process.env.NEXT_PUBLIC_BUILD_NUMBER}:`,
        // timeout for the Redis client operations like `get` and `set`
        // after this timeout, the operation will be considered failed and the `localCache` will be used
        timeoutMs: 5000,
      });
    } catch (error) {
      console.log(
        'Failed to initialize Redis cache, using local cache only.',
        error,
      );
    }
  }

  return {
    handlers: [redisCache, localCache],
    ttl: {
        // This value is also used as revalidation time for every ISR site
      defaultStaleAge: process.env.NEXT_PUBLIC_CACHE_IN_SECONDS, 
      // This makes sure, that resources without set revalidation time aren't stored infinitely in Redis
      estimateExpireAge: (staleAge) => staleAge, 
    },
  };
});

module.exports = CacheHandler;

But here is one interesting caveat. What if Redis isn’t available during server start? The line await client.connect(); will fail, and the page will load with a delay. But because of this, Next.js will try to initialize a new CacheHandler every time someone visits any page.

That is why we decided to use only LRU in such cases. However, the solution to this problem is not trivial, as createClient doesn’t throw errors; it operates only on callbacks. So a workaround is needed:

      ...
      let isReady = false;

      const client = createClient({
        url: process.env.REDIS_URL,
        socket: {
          reconnectStrategy: () => (isReady ? 5000 : false),
        },
      )};

      client.on('error', (error) => {
        console.error('Redis error', error);
      });

      client.on('ready', () => {
        isReady = true;
      });

      await client.connect();

      ...

This ensures that Next.js will not try to reconnect if the initial connection fails. In other cases, reconnection is desired and works like a charm.

Performance and stability

Our performance tests showed that CPU usage increased by about 2%, but response times stayed the same.

At Odrabiamy, our goal is not only to have a performant solution but also to have independent infrastructure layers, so that any failure does not influence the functioning of the entire application. This is where the Least Recently Used (LRU) cache comes into play as a crucial fallback mechanism. During our performance tests, we manually terminated Redis multiple times, which resulted in zero downtime. The transition between Redis and the LRU cache was so seamless that it wasn’t even noticeable in our performance graphs.

Conclusion

In the case of multiple Next.js instances running on a Kubernetes cluster, it is worth considering replacing the default file-system based cache with a Redis one. This can free up your storage resources without any risks and performance downgrades. Setting this configuration up is quite easy and was already battle-tested in our production environment.