Forem: Ahmed Atwa

[Boost]

Ahmed Atwa — Sat, 04 Apr 2026 12:48:09 +0000

Ahmed Atwa

Aug 21 '24

Why Docker is a Game-Changer for Development, Not Just Production

#webdev #docker #devops #productivity

Comments 5

7 min read

Why Docker is a Game-Changer for Development, Not Just Production

Ahmed Atwa — Wed, 21 Aug 2024 16:01:49 +0000

Introduction

In today’s fast-paced software development landscape, Docker is not just a tool for deployment but a transformative force in development workflows. While its role in streamlining production environments is well-known, Docker's impact on development is equally significant and often overlooked.

In this blog, we'll discover the hidden gems of using Docker for your everyday work and development. We'll go over the current problems in modern application development so that it will be clearer how Docker can solve these problems and make our lives less miserable. We will also look at practical examples of how to set up a Dockerized workflow in 2 different ways and compare them. By the end of this blog, you should be able to convince yourself or your manager why you should containerize your development workflow!

Let's dive into it!

Challenges in Traditional Development

Complex Setup and Configuration

Today, it has become a norm that onboarding new members to the team and installing and configuring the project may take a day or more, following a huge and (could be) poorly documented README, and a lot of debugging and StackOverflow-ing. I am not talking about simple Node.js apps that are simply installed via npm install, I am talking about huge applications and open-source projects, which will require you to install dozens of dependencies and tools just to run the project. I still remember going days over installing a particular open-source project and still couldn't get the tests to run properly...

Inconsistent Environments, Dependency Conflicts

Each developer has their own environment, software packages and dependencies installed. This leads to problems and conflicts when installing and running the project, leading to more debugging and wasted time.

Another major issue, what if you have a backend that runs on Node.js v16 and a frontend app that runs on Node.js v20? Which Node.js version will you install? Even if you use NVM (Node Version Manager), only 1 version of Node.js will be running at a time, meaning you can't run the backend and frontend together to test them.

Learning, Testing and Experimentation Challenges

How would you go about testing the full project stack? You would need to install all the projects and run them together. You're most likely to suffer from one of the previously mentioned problems.

Need to learn load balancing and test out a load balancer, or test out scaling your application? You would need to run multiple instances of your application. How difficult that is will depend on your runtime, but you will have to assign different ports manually.

Want to learn SQL databases or Redis? Go into huge installation steps.

At the end, you hope that all the new software dependencies and installations didn't mess up your system and impact your daily life usage. BTW, good luck uninstalling a project's dependencies.

Now to the solution!

How Docker can fix your life (Err, your development problems)

Eases Onboarding and Contribution to Complex Projects

After cloning a project, and with a single command docker compose up --watch (no matter how complex the project is), the project will just magically run. Zero installations or configurations needed. Onboarding new members or contributing to new projects has never been easier!

And as a bonus, all changes they make to the code will instantly reflect in the container, allowing for a seamless and effortless development process.

Simplifies README Files

No need to spend hours documenting how to run the project, just to find out that only you can understand what you wrote. The setup section of the README will now be as small as: "run docker compose up --watch to run the development workflow".

Ensures Environment Consistency, Simplifies Dependency Management, and Offers Portability

With Docker, you can ensure that all developers are running the same environment, the same software versions, and the same dependencies, no matter what OS or configuration they have. Develop everywhere, run everywhere.

Provides Isolation and Handles Conflicting Dependencies Securely

Remember the problem of having different Node.js versions? Now you can run them both in separate containers, each with their Node.js version, and they won't interfere with each other. Everything is isolated. Everyone is happy. Want to delete a project? Just docker compose down and that's it, no need to worry about uninstalling anything.

Enables Practical Learning and Experimentation

Want to learn a new technology or tool? Just pull a Docker image and run it. No need to install anything or worry about messing up your system. Could be a database, a cache, a load balancer, a message broker, etc.

Learning frontend and want some ready-made backend to test your frontend with? Just pull a Docker image and run it. Same for backend developers.

Want to learn system design and how to scale your application? Just run multiple instances of your application in separate containers. No need to worry about ports or configurations.

Facilitates Consistent Testing Across Versions

Want to test something on an older version of your project? Just keep track of your Docker images and run that older version. No need to rollback your code or worry about conflicting dependencies.

Want to test across different environments? Just run your project in different containers with different configurations.

Want to test your whole stack? Easy as well. If you have some Docker knowledge, you can do anything you want.

Let's get our hands dirty!

Setting up Dockerized Development Workflow

For this section, we will be working on a Node.js application. Assuming it runs on Node.js v16.

First step is making the Dockerfile.

Dockerfiles allow us to describe the installation and configuration steps that will be run to build the final usable image. This is the part where you install all the dependencies and tools.

# Dockerfile
FROM node:16.20.2-slim # Defines a Docker image to build on

WORKDIR /app # Creates a directory and switches to it, inside the image
COPY package*.json ./ # Copy files into the image

ENV PORT=3000 # Sets environment variables
EXPOSE $PORT # Exposes a PORT to the outside

RUN npm install --force # Runs an arbitrary command, like installing npm modules
COPY . . # Copies all files in the directory into the image
ENTRYPOINT [ "npm", "run" ] # Sets up the entry point of the image
CMD [ "dev:docker" ] # Sets up the command, which will act as an argument to the entry point in this case

dev:docker is a package.json script defined as nodemon server.js --legacy-watch.

Second, creating .dockerignore

Similar to .gitignore, we don't want to copy all files into the Docker image we are creating. For example, we shouldn't copy the node_modules or the secrets. The .dockerignore file helps us to specify what not to copy.

Ex:

node_modules
.git
.env

A very simple Dockerfile that first copies the package.json and package-lock.json, then installs the packages, and then copies the rest of the project.

Third, Creating the Docker Compose file

Docker Compose files allow us to specify how the Docker image should be run and the specifications of the Docker container. Here we specify how changes will propagate from our local machine to the container.

There are 2 ways for the Dockerized development workflow:

Using Volumes

The old way. You will mount your project directory to the container, so any changes you make in your project will be reflected in the container.

Docker Compose file:

services:
  backend:
    container_name: my-nodejs-app
    image: my-nodejs-app:dev
    build: .
    ports:
      - 3000:3000
    volumes:
      - ./:/app
      - NOT_USED:/app/node_modules

volumes:
  NOT_USED:

Notice the volumes section: In the first line, we are mounting the current directory to the /app directory in the container. However, we want to exclude the node_modules directory from the mounting, so we are mounting a volume that is not used to the node_modules directory in the container. The end result is that now any changes made in our project will be reflected in the container, except node_modules.

In other runtimes, you exclude the equivalent of node_modules directory in that runtime.

For this method to work, the command running must be a dev command; it should watch for changes and restart the server automatically. If you install a new dependency, you will need to manually rebuild the image.

You run it via docker compose up.

Using Develop Specification

The new way, we will use the new develop specification in the Docker Compose that was made exactly for this purpose.

Docker Compose file:

services:
  backend:
    container_name: my-nodejs-app
    image: my-nodejs-app:dev
    build: .
    ports:
      - 3000:3000
    develop:
      watch:
        - action: sync
          path: ./
          target: /app
          ignore:
            - node_modules
        - action: rebuild
          path: package.json

In develop.watch, we have 2 items that define what happens when we change a file in the project directory.

In each item, we define the action to take, the path to watch, and the target in the container. Actions can be sync, rebuild, or sync+restart.

Our first item tells Docker to sync any changed file in the current directory on our machine to the target path on the container, excluding node_modules. I am using sync here because I am using a dev command inside the container, but I could have also used a production command with sync+restart.

The second item tells Docker to rebuild the image and restart the container if we change the package.json file, like when we install a new dependency.

You run it via docker compose watch or docker compose up --watch.

Learn more here: https://docs.docker.com/reference/compose-file/develop/

You can already see why the new method is superior. Now we declare what we need, and Docker will do it for us. We did not need a workaround for excluding node_modules, we are not forced into using a dev command, and we can properly handle new dependencies without manual intervention.

With the Dockerfile, .dockerignore, and Docker Compose made, you are ready to start your development! All changes you make will be reflected in your application.

Anyone cloning your project and running docker compose up --watch will immediately start the development process without any additional steps.

Conclusion

With that, we have seen how Docker can transform your development workflow and make your life easier. We have seen the problems in traditional development and how Docker can solve them. We have seen the benefits of using Docker for development and how it can make your life easier. We have seen 2 ways to set up a Dockerized development workflow and compared them.

I would like to request, especially from open-source maintainers, to consider Dockerizing their projects. Instead of pages of installation steps, you can make the required Docker image and Docker Compose file and let the contributors run the project with a single command. This will make your project more accessible and easier to contribute to.

I hope you enjoyed this blog and learned something new!

Personalized Social Images: Enhancing User Profiles with Opengraph

Ahmed Atwa — Wed, 24 May 2023 10:15:32 +0000

Hello! I am Ahmed Atwa, a Software Engineer and OpenSauced contributor, and today I'll be showcasing a backend project I worked on when I was working for OpenSauced, and open-source it by the end of the blog!

Origin of the idea

It all started when we noticed that many social media platforms have boring and generic profile preview images for their users. We wanted to spice things up and make them more personalized and fun. So we decided to create a service that generates social images based on your name, repositories, used languages, and achievements. You can use these images to showcase your personality and skills on your profile or on your highlights when sharing them through social media.

It was my task to create and manage this new backend service from Proof of Concept (POC) to production & connecting with the frontend.

Architecture

This is the tech stack we used:

Satori (for the image generation),
satori-html (to replace the need of React),
Resvg (to convert SVG into image)
NestJs & TypeScript

Proof of Concept

Before officially starting on it, we needed a proof of concept to determine what's needed and the potential problems we could face. I threw in a quick nodejs/express app that should just create a dummy image.

During the process, we found two key challenges:

The first problem we faced was that Satori library was made for React environments and required React elements as input. Luckily, we discovered satori-html which would allow the input to be a string (in the form of HTML) instead.
The second problem was with satori-html itself. The project is on CommonJS module mode and the package was on ECMAScript module mode. This was pretty tiresome to solve, but in the end the final solution was:

const html = async (...args: string[]) => {
  const { html } = await import('satori-html');
  // @ts-ignore
  return html(...args);
}

and setting "moduleResolution": "node16".

After solving these 2 problems, the proof of concept was done and it was time for the next step.

import fs from 'fs';
// @ts-ignore
import satori from 'satori'
import { Resvg } from '@resvg/resvg-js'
import HTMLTemplate from './HTMLTemplate';

const html = async (...args: string[]) => {
  const { html } = await import('satori-html');
  // @ts-ignore
  return html(...args);
}

export default async function createImage(name: string) {
  const template = await html(HTMLTemplate(name))

  let robotoArrayBuffer = fs.readFileSync('public/Roboto-Regular.ttf');
  const svg = await satori(template , {
      width: 600,
      height: 400,
      fonts: [
        {
          name: 'Roboto',
          data: robotoArrayBuffer,
          weight: 400,
          style: 'normal',
        },
      ],
    },
  )

  const resvg = new Resvg(svg, {
    background: "rgba(238, 235, 230, .9)",
  });

  const pngData = resvg.render()
  const pngBuffer = pngData.asPng()

  return pngBuffer
}

More technical context behind the decisions made is available here

Opengraph Repo

With the POC tested and approved, it was the moment for this creation to rise and shine. I worked with @0vortex on the Opengraph repo, I focused on the frontend side of the service, i.e. the card generation process, the card design, testing, etc., while he focused on the backend side of the service, basically setting up the infrastructure, fetching data from GitHub with GraphQL, the caching and storage of the cards, etc.

User Cards

The first card we made displayed a user's name, image, top repos, and top languages. The card was used on Insights user profile page.

Highlight Cards

The second card we developed was the highlight card, which provided information about a specific highlight, including the owner's name, image, highlight's body, project details, languages used, and reactions count.

The Process from Start to Finish

I'll be talking specifically about User cards process, which involved two endpoints: one for generation and another for status checking.

The status endpoint is the initial step, allowing us to determine whether the card is up-to-date, out-dated, or non-existent. In all cases, a URL is provided to locate the card.
The generation endpoint internally checks the status first. If the card is up-to-date, it redirects to the card URL. If not, the card is re-generated and uploaded to the cloud.

Problem with Testing

We focused on getting the project ready for production first, and there was a little problem; we had no way to locally test the cards, without activating the whole process of generating and uploading to the cloud.

I thought of making some checks on the environment whether it's production or local and branched the behavior depending on the environment, but this would have turned the straight-forward code into a big complex spaghetti, and would've been very hard to maintain.

To address this challenge, I devised a solution that involved separate node scripts that would activate only the generation functions and generate images in a local folder. Now I wouldn't have to mess with the production process and keep everything simple and elegant. These new node scripts could be called separately without running the server.

This is the node script for testing user profiles: local-dev/UserCards.ts

const testUsernames = [
  "bdougie", "deadreyo", "defunkt", "0-vortex", "Anush008", "diivi"
];

const folderPath = "dist";

async function testUserCards () {
  const moduleFixture: TestingModule = await Test.createTestingModule({ imports: [AppModule] }).compile();

  const app = moduleFixture.createNestApplication();

  await app.init();

  const instance = app.get(UserCardService);

  const promises = testUsernames.map(async username => {
    const { svg } = await instance.generateCardBuffer(username);

    if (!existsSync(folderPath)) {
      await mkdir(folderPath);
    }
    await writeFile(`${folderPath}/${username}.svg`, svg);
  });

  // generating sequential: 10.5 seconds, parallel: 4.5 seconds
  await Promise.all(promises);
}

testUserCards();

A common pattern I made in testing and when fetching the image on Insight frontend was to run promises in parallel, to speed up the process by 100%.

And with this, the Opengraph was done and it was time to integrate the whole thing into the User Profiles on Insights!

Insights User Profiles

By default, Next.js uses SSG (Static Site Generation), and this was simply not suitable for our purposes. If we wanted dynamic user profiles with dynamic social cards, then it was a must to turn User Profiles into SSR (Server Side Rendering). For a detailed explanation, please refer to this link.

The process involved two main steps:

The first step was turning it SSR. This activated the meta-tags for the profiles which were installed but not running due to SSG.
The second step was fetching the image from Opengraph. First we hit the metadata endpoint to check the status of the image, if OK, get the card URL (sent in the response headers) and insert it into the meta-tags. If not, generate it and still insert the location to meta-tags. (Cards URLs are static, so even if the image isn't there, we know where it will be.)

Note: An issue with this is that the URL preview may be displayed before the card finishes generation, leading to a broken preview image. We chose to ignore this issue for the time being. (It would break only on the first time anyways.)

This is the SSR code:

export const getServerSideProps = async (context: UserSSRPropsContext) => {
  return await handleUserSSR(context);
};

export type UserSSRPropsContext = GetServerSidePropsContext<{ username: string }>;

export async function handleUserSSR({ params }: GetServerSidePropsContext<{ username: string }>) {
  const { username } = params!;
  const sessionResponse = await supabase.auth.getSession();
  const sessionToken = sessionResponse?.data.session?.access_token;
  let user;
  let ogImage;

  async function fetchUserData() {
    const req = await fetch(`${process.env.NEXT_PUBLIC_API_URL}/users/${username}`, {
      headers: {
        accept: "application/json",
        Authorization: `Bearer ${sessionToken}`
      }
    });

    user = await req.json() as DbUser;
  }

  async function fetchSocialCardURL() {
    const socialCardUrl = `${String(process.env.NEXT_PUBLIC_OPENGRAPH_URL ?? "")}/users/${username}`;
    const ogReq = await fetch(`${socialCardUrl}/metadata`); //status returned: 204 or 304 or 404
    if(ogReq.status !== 204) {
      fetch(socialCardUrl, {
        method: "HEAD"
      }); // trigger the generation of the social card
    }

    ogImage = ogReq.headers.get("x-amz-meta-location");
  }

  // Runs the data fetching in parallel. Decreases the loading time by 50%.
  await Promise.allSettled([fetchUserData(), fetchSocialCardURL()]);

  return {
    props: { username, user, ogImage }
  };
}

Separated the contents of the getServerSideProps into another function & created UserSSRPropsContext as I would need to duplicate the code into 2 more pages/links that basically point to the same page. Now I don't need to touch the other 2 ever.

Another thing was running data fetching in parallel, since each one is independent of the other, then we can save time. (Saving time is critical since this is Server-side rendering, i.e. user waits for the server.)

Conclusion

This was the whole story behind getting Opengraph repo from ground till production & integrating with the frontend!
I really enjoyed working on this project very much and it's only the beginning of my journey! I'm looking forward to making future contributions ✨

The project is about to start a new adventure, as an Open Source project!

https://github.com/open-sauced/opengraph.opensauced.pizza