Forem: Damir Mahamedov

Setting Up PostgreSQL in Docker on Hetzner with SSL Certificates from Certbot

Damir Mahamedov — Mon, 29 Sep 2025 18:40:47 +0000

Securing a PostgreSQL instance with SSL is essential for protecting your data, especially when deploying on cloud platforms like Hetzner. In this guide, we’ll set up PostgreSQL in Docker, configure SSL using Certbot certificates, and automate certificate renewal using a cron job to ensure minimal downtime for your database.

Prerequisites

Before proceeding, ensure you have:

A Hetzner cloud server running a Linux distribution (e.g., Ubuntu).
Docker and Docker Compose installed on your server.
A domain name (e.g., db.example.com) pointing to your Hetzner server.
Certbot installed for generating SSL certificates.

Hetzner Firewall Configuration

To ensure your server is accessible and secure, you need to configure the Hetzner firewall to open the necessary inbound ports. Here’s what you need to allow:

Port 80 (HTTP): Required for Certbot to verify your domain when issuing SSL certificates.
Port 22 (SSH): Required for SSH access to your server.
Port 5432 (PostgreSQL): Required for PostgreSQL clients to connect to the database.

You also need to allow TCP outbound traffic on any port, which is necessary for network connectivity.

Configuring Hetzner Firewall

Log in to your Hetzner Cloud Console.
Go to the Firewall section and create a new firewall.
Add the following inbound rules:
- HTTP (Port 80): To allow Certbot to perform domain validation.
- SSH (Port 22): To allow SSH access to your server.
- PostgreSQL (Port 5432): To allow connections to your PostgreSQL database (from your IP or trusted IPs).
Add the following outbound rule:
- TCP (any port): To allow your server to communicate freely over the internet, which is needed for Certbot renewal and other outbound communications.
Assign the firewall to your server.

Why Use SSL with PostgreSQL?

SSL (Secure Sockets Layer) ensures that data transmitted between PostgreSQL and its clients is encrypted. It protects against potential man-in-the-middle attacks and data theft by encrypting the communication channel.

Step 1: Install Docker and Docker Compose

If Docker and Docker Compose aren’t already installed on your Hetzner server, follow these steps.

Installing Docker

SSH into your Hetzner server.
Install Docker:

sudo apt update
sudo apt install docker.io -y

Enable Docker to start on boot and verify the installation:

sudo systemctl start docker
sudo systemctl enable docker

Installing Docker Compose

Install Docker Compose:

sudo apt install docker-compose -y

Step 2: Install Certbot and Generate SSL Certificates

Certbot simplifies generating SSL certificates from Let’s Encrypt. Here’s how to install and generate SSL certificates:

Install Certbot:

sudo apt install certbot -y

Generate SSL certificates for your domain:

sudo certbot certonly --standalone -d db.example.com

The certificates will be saved in /etc/letsencrypt/live/db.example.com/. The important files for PostgreSQL are:

fullchain.pem (public certificate)
privkey.pem (private key)

Automate Certificate Renewal Using Cron

Let’s Encrypt certificates expire every 90 days, so it’s crucial to set up automated certificate renewal. Certbot can handle this automatically using cron, ensuring that your SSL certificates are renewed without manual intervention.

Open the cron file:

sudo crontab -e

Add the following cron job to renew the certificates and restart the PostgreSQL container:

0 3 * * * /usr/bin/certbot renew --quiet && docker-compose -f /path/to/your/docker-compose.yml restart postgres

Explanation:

0 3 * * *: This cron job will run every day at 3:00 AM. Adjust the time as needed.
/usr/bin/certbot renew --quiet: This runs Certbot’s renewal process in quiet mode, meaning it will renew certificates only if they are about to expire (i.e., within 30 days of expiration).
docker-compose -f /path/to/your/docker-compose.yml restart postgres: After renewal, this command restarts the PostgreSQL container to apply the new SSL certificates.

Save and exit the cron file. Now, Certbot will automatically renew the certificates, and PostgreSQL will restart to load the new certificates with no downtime for your application.

Step 3: Configure Docker Compose for PostgreSQL

Now we’ll configure PostgreSQL using Docker Compose, mounting SSL certificates, and setting up environment variables.

The docker-compose.yml File

Create a docker-compose.yml file to define the PostgreSQL service:

nano docker-compose.yml

Here’s the configuration:

services:
  postgres:
    image: postgres:16
    container_name: postgres_db
    restart: always
    env_file:
      - ./.env
    environment:
      - POSTGRES_USER=$DB_USERNAME
      - POSTGRES_PASSWORD=$DB_PASSWORD
      - POSTGRES_DB=$DB_NAME
    volumes:
      - ./pg_data:/var/lib/postgresql/data
      - ./pg_hba.conf:/var/lib/postgresql/data/pg_hba.conf
      - /etc/letsencrypt/live/db.example.com/fullchain.pem:/var/lib/postgresql/fullchain.pem
        - /etc/letsencrypt/live/db.example.com/privkey.pem:/var/lib/postgresql/privkey.pem
    networks:
      - db-net
    ports:
      - "5432:5432"
    command: >
      postgres -c shared_buffers=2GB
               -c work_mem=16MB
               -c maintenance_work_mem=256MB
               -c effective_cache_size=6GB
               -c ssl=on
               -c ssl_cert_file='/var/lib/postgresql/fullchain.pem'
               -c ssl_key_file='/var/lib/postgresql/privkey.pem'
               -c listen_addresses='*'

volumes:
  pg_data:

networks:
  db-net:
    driver: bridge

Explanation of Memory and Performance Options

-c shared_buffers=2GB
The shared_buffers setting determines how much memory PostgreSQL will use to cache frequently accessed data. Setting this higher can significantly improve performance by reducing disk I/O since the data will be stored in memory instead of being read from disk repeatedly.

Default: The default value is usually very low, often 128MB or 256MB, which is insufficient for production systems.

Recommended Setting: A general recommendation is to allocate about 25–40% of the total available RAM to shared_buffers. In this example, 2GB has been allocated, which is suitable for systems with at least 8GB of RAM.
-c work_mem=16MB
The work_mem setting specifies the amount of memory allocated for each query operation that requires memory (e.g., sorting, hash tables). Each connection and each query can use its own chunk of work_mem.

Default: Typically around 4MB.

Recommended Setting: Increasing this to 16MB allows queries that involve sorting or creating hash tables to perform faster by reducing the need to write temporary data to disk.

Note: Be cautious with this setting, as high values combined with many concurrent users can lead to excessive memory usage.
-c maintenance_work_mem=256MB
The maintenance_work_mem setting controls the memory used for maintenance tasks such as VACUUM, CREATE INDEX, and ALTER TABLE. These operations can be memory-intensive, and allocating more memory here can speed up these operations.

Default: Often set around 64MB.

Recommended Setting: For production environments, especially those with large tables, increasing this to 256MB helps PostgreSQL complete maintenance tasks faster.
-c effective_cache_size=6GB
The effective_cache_size parameter is a guideline that PostgreSQL uses to estimate the amount of memory available for disk caching by the operating system. This setting doesn’t allocate memory but helps PostgreSQL make decisions about which queries to optimize and which indexes to use.

Default: Often set to 4GB.

Recommended Setting: The value should generally be about 50–75% of the system’s total memory. In this case, with 6GB, PostgreSQL will assume there is enough memory to cache large chunks of data, leading to more efficient queries.

The .env File

Create a .env file in the same directory as your docker-compose.yml:

nano .env

Add the following environment variables:

DB_USERNAME=your_db_user
DB_PASSWORD=your_secure_password
DB_NAME=your_database_name

Step 4: Configure `pg_hba.conf` to Restrict Access by IP Address

The pg_hba.conf file is used to control access to PostgreSQL. In this step, we’ll configure it to only allow connections from a specific IP address (99.99.99.99) using SSL and block all other IP addresses.

Create the pg_hba.conf file:

nano pg_hba.conf

Add the following configuration:

# Allow only the specific IP address 99.99.99.99 to connect via SSL
hostssl    all             all             99.99.99.99/32      scram-sha-256

# Block all other IP addresses
hostssl    all             all             0.0.0.0/0           reject

Explanation:

hostssl: Enforces SSL for all connections.
99.99.99.99/32: Allows only connections from the IP address 99.99.99.99.
scram-sha-256: Uses SCRAM-SHA-256 authentication for secure password hashing. Please don’t use md5.
0.0.0.0/0: Blocks all other IP addresses from accessing the database.

Step 5: Adjust File Permissions for SSL Certificates

PostgreSQL requires strict permissions on the private key file (privkey.pem). It must be owned by the postgres user inside the Docker container, and the permissions must be set to 0600 (read/write only for the owner).

Set the appropriate permissions on the host machine:

sudo chown 999:999 /etc/letsencrypt/live/db.goachievo.com/privkey.pem
sudo chmod 600 /etc/letsencrypt/live/db.goachievo.com/privkey.pem

PostgreSQL also needs read access to the public certificate file:

sudo chown 999:999 /etc/letsencrypt/live/db.goachievo.com/fullchain.pem
sudo chmod 644 /etc/letsencrypt/live/db.goachievo.com/fullchain.pem

Why 999?
Inside the Docker container, the postgres user typically has the UID and GID of 999. This ensures that PostgreSQL can read the private key.

Step 6: Start PostgreSQL

With everything set up, start the PostgreSQL container:

docker-compose up -d

Docker will pull the PostgreSQL image, create the necessary volumes, and start the database with SSL enabled and access restricted to the specified IP address.

Step 7: Verify SSL and Connection Restrictions

Once PostgreSQL is running, verify that SSL is enabled and that the connection restrictions are in place.

Connect to the PostgreSQL instance and run:

SHOW ssl;

It should return on, indicating that SSL is enabled.

Check the PostgreSQL logs to verify SSL connections and confirm that only the allowed IP address is connecting:
docker logs postgres_db

Test the connection from the allowed IP (99.99.99.99) and ensure that connections from other IP addresses are blocked.

Conclusion

In this article, we’ve successfully set up PostgreSQL in Docker with SSL certificates from Certbot, restricted access to a specific IP address using pg_hba.conf, and ensured secure encrypted connections. By using cron for automated certificate renewal, you’ve also ensured that your SSL certificates remain up to date with minimal manual intervention.

Feel free to share your thoughts and improvements in the comments section below!

If you enjoyed this article or found these tools useful, make sure to follow me on Medium for more insights and tips on coding and development. I regularly share helpful content to make your coding journey smoother.

Follow me on X (Twitter), where I share more interesting thoughts, updates, and discussions about programming and tech! Don’t miss out — click those follow buttons.

You can also follow me on LinkedIn for professional insights, updates on my latest projects, and discussions about coding, tech trends, and more. Don’t miss out on valuable content that can help you level up your development skills — let’s connect!

5 Tools to Turn Your Code into Beautiful Visuals for Sharing

Damir Mahamedov — Mon, 07 Oct 2024 08:05:24 +0000

As developers, we often need to share code — whether it’s to help others learn, demonstrate our projects, or simply show off a solution we’re proud of. However, raw code doesn’t always look great in a tweet or blog post. Thankfully, there are some fantastic tools that can help turn our code into visually appealing images that are easier to digest and share.

In this article, I’ll walk you through 5 powerful tools that allow you to create professional-looking code snippets for sharing on social media or in presentations.

1. Carbon

Website: carbon.now.sh

Why It’s Great: Carbon is one of the most popular tools for creating and sharing beautiful code snippets. It’s widely used by developers on platforms like X (formerly Twitter) and GitHub due to its flexibility and range of customization options.

Features:

Supports multiple languages, including JavaScript, Python, and Go.
Dozens of customizable themes.
Ability to adjust window style, font, and background color.
Easily export images as PNG or SVG files.

How to Use It:

Go to Carbon and paste your code.
Customize the appearance (themes, background, window controls).
Export your image and share it on your platform of choice.

Pro Tip: Try the “Dracula” or “Solarized Dark” theme for a modern, eye-catching look!

2. Ray.so

Website: ray.so

Why It’s Great: Ray.so is perfect for developers who want to create dark mode code snippets that stand out. It’s simple to use and offers a sleek design with minimal configuration. This is a great tool if you need to create a quick, clean visual without too many customizations.

Features:

Dark mode by default.
Simple and intuitive UI.
Options for adjusting the padding and text size.
Exports in high-quality PNG format.

How to Use It:

Visit Ray.so, paste your code, and select your preferred settings.
Adjust the padding and color if needed.
Download your snippet and you’re ready to share!

3. Snappify

Website: snappify.io

Why It’s Great: Snappify stands out because of its extensive customization options. Beyond just changing the background color or font, Snappify allows you to adjust the layout, add annotations, and even create multi-snippet visuals. It’s perfect for educational content, tutorials, or explaining complex ideas.

Features:

Full control over layout and design.
Ability to add annotations, comments, or diagrams.
Drag-and-drop support for arranging snippets.
Multiple output formats including PNG and SVG.

How to Use It:

Go to Snappify, paste your code, and use the layout editor to organize it.
Add annotations if needed (great for explaining the code).
Export your snippet and post it wherever you need.

Pro Tip: Snappify is great for creating infographics or X (Twitter) threads that require more explanation.

4. Polacode (VS Code Extension)

Website: Polacode

Why It’s Great: Polacode is perfect for VS Code users who want to quickly turn their code into an image without leaving the editor. It integrates directly with VS Code, making it highly convenient if you’re already working on a project and want to share snippets without switching tools.

Features:

Direct integration into VS Code.
Supports all languages that VS Code supports.
Easily customizable (theme, font, and layout).
Exports to PNG.

How to Use It:

Install Polacode from the VS Code Marketplace.
Highlight the code you want to capture and open Polacode.
Customize the appearance, then export your image.

Pro Tip: Since Polacode uses your current VS Code theme, make sure you’re using a visually appealing theme like “Monokai” or “One Dark Pro” for polished results.

5. Codeimg.io

Website: codeimg.io

Why It’s Great: Codeimg.io is a simple, no-frills tool for creating quick code snippets. It’s great if you need a basic visual without too many customization options. The interface is easy to use, and you can go from raw code to an image in seconds.

Features:

Support for multiple themes and languages.
Simple UI for fast snippet creation.
Export to PNG, JPEG, or SVG.

How to Use It:

Go to Codeimg.io, paste your code, and select a theme.
Adjust the font size, color, and padding.
Download your snippet and you’re good to go.

Conclusion

Whether you’re looking to share a quick snippet on Twitter, showcase a new feature, or teach a concept, using one of these tools will help make your code more engaging and easier to understand. Each tool has its own unique strengths, so choose the one that fits your style and workflow the best.

What’s Your Favorite Tool? If you have a favorite tool that I didn’t mention, let me know in the comments! I’d love to hear how you share your code with the world.

Feel free to share your thoughts and improvements in the comments section below!

Stay Connected for More Insights

Medium for coding tips and insights.
X (Twitter) for programming thoughts and tech updates.
LinkedIn for professional insights and project updates.
Instagram for behind-the-scenes looks at my coding journey.

Streamline File Uploads in NestJS: Efficient In-Memory Parsing for CSV & XLSX Without Disk Storage

Damir Mahamedov — Mon, 30 Sep 2024 16:56:54 +0000

Effortless File Parsing in NestJS: Manage CSV and XLSX Uploads in Memory for Speed, Security, and Scalability

Introduction

Handling file uploads in a web application is a common task, but dealing with different file types and ensuring they are processed correctly can be challenging. Often, developers need to parse uploaded files without saving them to the server, which is especially important for reducing server storage costs and ensuring that sensitive data is not unnecessarily retained. In this article, we’ll walk through the process of creating a custom NestJS module to handle file uploads specifically for CSV and XLS/XLSX files, and we’ll parse these files in memory using Node.js streams, so no static files are created on the server.

Why NestJS?

NestJS is a progressive Node.js framework that leverages TypeScript and provides an out-of-the-box application architecture that enables you to build highly testable, scalable, loosely coupled, and easily maintainable applications. By using NestJS, we can take advantage of its modular structure, powerful dependency injection system, and extensive ecosystem.

Step 1: Setting Up the Project

Before we dive into the code, let’s set up a new NestJS project. If you haven’t already, install the NestJS CLI:

npm install -g @nestjs/cli

Create a new NestJS project:

nest new your-super-name

Navigate into the project directory:

cd your-super-name

Step 2: Installing Required Packages

We’ll need to install some additional packages to handle file uploads and parsing:

npm install @nestjs/platform-express multer exceljsfile-type

Multer: A middleware for handling multipart/form-data, which is primarily used for uploading files.
Exlesjs: A powerful library for parsing CSV/XLS/XLSX files.
File-Type: A library for detecting the file type of a stream or buffer.

Step 3: Creating the Multer Storage Engine Without Saving Files

To customize the file upload process, we’ll create a custom Multer storage engine. This engine will ensure that only CSV and XLS/XLSX files are accepted, parse them in memory using Node.js streams, and return the parsed data without saving any files to disk.

Create a new file for our engine:

import { PassThrough } from 'stream';
import * as fileType from 'file-type';
import { BadRequestException } from '@nestjs/common';
import { Request } from 'express';
import { Workbook } from 'exceljs';
import { createParserCsvOrXlsx } from './parser-factory.js';

const ALLOWED_MIME_TYPES = [
  'text/csv',
  'application/vnd.ms-excel',
  'text/comma-separated-values',
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
  'application/vnd.ms-excel',
] as const;

export class CsvOrXlsxMulterEngine {
  private destKey: string;
  private maxFileSize: number;
  constructor(opts: { destKey: string; maxFileSize: number }) {
    this.destKey = opts.destKey;
    this.maxFileSize = opts.maxFileSize;
  }
  async _handleFile(req: Request, file: any, cb: any) {
    try {
      const contentLength = Number(req.headers['content-length']);
      if (
        typeof contentLength === 'number' &&
        contentLength > this.maxFileSize
      ) {
        throw new Error(`Max file size is ${this.maxFileSize} bytes.`);
      }
      const fileStream = await fileType.fileTypeStream(file.stream);
      const mime = fileStream.fileType?.mime ?? file.mimetype;
      if (!ALLOWED_MIME_TYPES.includes(mime)) {
        throw new BadRequestException('File must be *.csv or *.xlsx');
      }
      const replacementStream = new PassThrough();
      fileStream.pipe(replacementStream);
      const parser = createParserCsvOrXlsx(mime);
      const data = await parser.read(replacementStream);
      cb(null, {
        [this.destKey]:
          mime === 'text/csv' ? data : (data as Workbook).getWorksheet(),
      });
    } catch (error) {
      cb(error);
    }
  }
  _removeFile(req: Request, file: any, cb: any) {
    cb(null);
  }
}

This custom storage engine checks the file’s MIME type and ensures it’s either a CSV or XLS/XLSX file. It then processes the file entirely in memory using Node.js streams, so no temporary files are created on the server. This approach is both efficient and secure, especially when dealing with sensitive data.

Step 4: Creating the Parser Factory

The parser factory is responsible for determining the appropriate parser based on the file type.

Create a new file for our parser:

import excel from 'exceljs';

export function createParserCsvOrXlsx(mime: string) {
  const workbook = new excel.Workbook();
  return [
    'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
    'application/vnd.ms-excel',
  ].includes(mime)
    ? workbook.xlsx
    : workbook.csv;
}

This factory function checks the MIME type and returns the appropriate parser (either xlsx or csv).

Step 5: Configuring Multer in the NestJS Controller

Next, let’s create a controller to handle file uploads using our custom storage engine.

Generate a new controller:

nest g controller files

In the files.controller.ts, configure the file upload using Multer and the custom storage engine:

import {
  Controller,
  Post,
  UploadedFile,
  UseInterceptors,
} from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { Worksheet } from 'exceljs';
import { CsvOrXlsxMulterEngine } from '../../shared/multer-engines/csv-xlsx/engine.js';
import { FilesService } from './files.service.js';

const MAX_FILE_SIZE_IN_MiB = 1000000000; // Only for test

@Controller('files')
export class FilesController {
  constructor(private readonly filesService: FilesService) {}
  @UseInterceptors(
    FileInterceptor('file', {
      storage: new CsvOrXlsxMulterEngine({
        maxFileSize: MAX_FILE_SIZE_IN_MiB,
        destKey: 'worksheet',
      }),
    }),
  )
  @Post()
  create(@UploadedFile() data: { worksheet: Worksheet }) {
    return this.filesService.format(data.worksheet);
  }
}

This controller sets up an endpoint to handle file uploads. The uploaded file is processed by the CsvOrXlsxMulterEngine, and the parsed data is returned in the response without ever being saved to disk.

Step 6: Setting Up the Module

Finally, we need to set up a module to include our controller.

Generate a new module:

nest g module files

In the files.module.ts, import the controller:

import { Module } from '@nestjs/common';
import { FilesController } from './files.controller.js';
import { FilesService } from './files.service.js';

@Module({
  providers: [FilesService],
  controllers: [FilesController],
})
export class FilesModule {}

Make sure to import this module into your AppModule:

Step 7: Testing the File Upload with HTML

To test the file upload functionality, we can create a simple HTML page that allows users to upload CSV or XLS/XLSX files. This page will send the file to our /api/files endpoint, where it will be parsed and processed in memory.

Here’s the basic HTML file for testing the file upload:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>File Upload</title>
</head>
<body>
    <h1>Upload a File (CSV or XLSX)</h1>
    <form action="/api/files" method="post" enctype="multipart/form-data">
        <label for="file">Choose file:</label>
        <input type="file" id="file" name="file" accept=".csv, .xlsx" required>
        <br><br>
        <button type="submit">Upload</button>
    </form>
</body>
</html>

To render the HTML page for file uploads, we first need to install an additional NestJS module called @nestjs/serve-static. You can do this by running the following command:

npm install @nestjs/serve-static

After installing, we need to configure this module in AppModule:

import { Module } from '@nestjs/common';
import { join } from 'path';
import { ServeStaticModule } from '@nestjs/serve-static';
import { FilesModule } from './modules/files/files.module.js';

@Module({
  imports: [
    FilesModule,
    ServeStaticModule.forRoot({
      rootPath: join(new URL('..', import.meta.url).pathname, 'public'),
      serveRoot: '/',
    }),
  ],
})
export class AppModule {}

This setup will allow us to serve static files from the public directory. Now, we can open the file upload page by navigating to http://localhost:3000 in your browser.

Upload Your File

To upload a file, follow these steps:

Choose a file by clicking on the ‘Choose file’ button.
Click on the ‘Upload’ button to start the upload process.

Once the file is uploaded successfully, you should see a confirmation that the file has been uploaded and formatted.

Note: I haven’t included code for formatting the uploaded file, as this depends on the library you choose for processing CSV or XLS/XLSX files. You can view the complete implementation on GitHub.
Comparing Pros and Cons of In-Memory File Processing
When deciding whether to use in-memory file processing or saving files to disk, it’s important to understand the trade-offs.

Pros of In-Memory Processing:

No Temporary Files on Disk:

Security: Sensitive data isn’t left on the server’s disk, reducing the risk of data leaks.
Resource Efficiency: The server doesn’t need to allocate disk space for temporary files, which can be particularly useful in environments with limited storage.

Faster Processing:

Performance: Parsing files in memory can be faster since it eliminates the overhead of writing and reading files from disk.
Reduced I/O Operations: Fewer disk I/O operations means lower latency and potent ially higher throughput for file processing.

Simplified Cleanup:

No Cleanup Required: Since files aren’t saved to disk, there’s no need to manage or clean up temporary files, simplifying the codebase.

Cons of In-Memory Processing:

Memory Usage:

High Memory Consumption: Large files can consume significant amounts of memory, which might lead to out-of-memory errors if the server doesn’t have enough resources.
Scalability: Handling large files or multiple file uploads simultaneously may require careful memory management and scaling strategies.

File Size Limitations:

Limited by Memory: The maximum file size that can be processed is limited by the available memory on the server. This can be a signific ant drawback for applications dealing with very large files.

Complexity in Error Handling:

Error Management: Managing errors in streaming data can be more complex than handling files on disk, especially in cases where partial data might need to be recovered or analyzed.

When to Use In-Memory Processing:

Small to Medium Files: If your application deals with relatively small files, in-memory processing can offer speed and simplicity.

Security-Sensitive Applications: When handling sensitive data that shouldn’t be stored on disk, in-memory processing can reduce the risk of data breaches.

High-Performance Scenarios: Applications that require high throughput and minimal latency may benefit from the reduced overhead of in-memory processing.

When to Consider Disk-Based Processing:

Large Files: If your application needs to process very large files, disk-based processing may be necessary to avoid running out of memory.

Resource-Constrained Environments: In cases where server memory is limited, processing files on disk can prevent memory exhaustion and allow for better resource management.

Persistent Storage Needs: If you need to retain a copy of the uploaded file for auditing, backup, or later retrieval, saving files to disk is necessary.

Integration with External Storage Services: For large files, consider uploading them to external storage services like AWS S3, Google Cloud

Storage, or Azure Blob Storage. These services allow you to offload storage from your server, and you can process the files in the cloud or retrieve them for in-memory processing as needed.

Scalability: Cloud storage solutions can handle massive files and provide redundancy, ensuring that your data is safe and easily accessible from multiple geographic locations.

Cost Efficiency: Using cloud storage can be more cost-effective for handling large files, as it reduces the need for local server resources and provides pay-as-you-go pricing.

Conclusion

In this article, we’ve created a custom file upload module in NestJS that handles CSV and XLS/XLSX files, parses them in memory, and returns the parsed data without saving any files to disk. This approach leverages the power of Node.js streams, making it both efficient and secure, as no temporary files are left on the server.

We’ve also explored the pros and cons of in-memory file processing versus saving files to disk. While in-memory processing offers speed, security, and simplicity, it’s important to consider the memory usage and potential file size limitations before adopting this approach.

Whether you’re building an enterprise application or a small project, handling file uploads and parsing correctly is crucial. With this setup, you’re well on your way to mastering file uploads in NestJS without worrying about unnecessary server storage or data security issues.

Feel free to share your thoughts and improvements in the comments section below!

If you enjoyed this article or found these tools useful, make sure to follow me on Dev.to for more insights and tips on coding and development. I regularly share helpful content to make your coding journey smoother.

Follow me on X (Twitter), where I share more interesting thoughts, updates, and discussions about programming and tech! Don't miss out - click those follow buttons.

You can also follow me on LinkedIn for professional insights, updates on my latest projects, and discussions about coding, tech trends, and more. Don't miss out on valuable content that can help you level up your development skills - let's connect!

Forem: Damir Mahamedov

Setting Up PostgreSQL in Docker on Hetzner with SSL Certificates from Certbot

Prerequisites

Hetzner Firewall Configuration

Configuring Hetzner Firewall

Why Use SSL with PostgreSQL?

Step 1: Install Docker and Docker Compose

Installing Docker

Installing Docker Compose

Step 2: Install Certbot and Generate SSL Certificates

Automate Certificate Renewal Using Cron

Step 3: Configure Docker Compose for PostgreSQL

The docker-compose.yml File

Explanation of Memory and Performance Options

The .env File

Step 4: Configure pg_hba.conf to Restrict Access by IP Address

Step 5: Adjust File Permissions for SSL Certificates

Step 6: Start PostgreSQL

Step 7: Verify SSL and Connection Restrictions

Conclusion

5 Tools to Turn Your Code into Beautiful Visuals for Sharing

1. Carbon

2. Ray.so

3. Snappify

4. Polacode (VS Code Extension)

5. Codeimg.io

Conclusion

Stay Connected for More Insights

Streamline File Uploads in NestJS: Efficient In-Memory Parsing for CSV & XLSX Without Disk Storage

Introduction

Why NestJS?

Step 1: Setting Up the Project

Step 2: Installing Required Packages

Step 3: Creating the Multer Storage Engine Without Saving Files

Step 4: Creating the Parser Factory

Step 5: Configuring Multer in the NestJS Controller

Step 6: Setting Up the Module

Step 7: Testing the File Upload with HTML

Pros of In-Memory Processing:

Cons of In-Memory Processing:

When to Use In-Memory Processing:

When to Consider Disk-Based Processing:

Conclusion

Step 4: Configure `pg_hba.conf` to Restrict Access by IP Address