Forem: Ninjeneer

For anyone suffering from Out Of Memory errors at build time in Vercel, due to Sentry ✅

Ninjeneer — Mon, 01 Sep 2025 14:03:48 +0000

Ninjeneer

Sep 1 '25

Preventing Vercel "Out of Memory" build errors

#vercel #sentry #nextjs #githubactions

Comments

4 min read

Preventing Vercel "Out of Memory" build errors

Ninjeneer — Mon, 01 Sep 2025 14:01:06 +0000

Vercel is a great platform when it comes to hosting easily any web projects. Using it with their own product, NextJS, makes it even smoother. It allows any developer to be fully production-ready in such a short time, and for free : plug your Github repository, do some quick setup and boom, you're live 🚀

However, when your project is growing and scaling, you start to hit Vercel limitations, at least, free tier limitations.

Context

Here at Fluum we are building the next-gen AI Co-Founder for solopreneurs, allowing anyone to run their business and sell their services seamlessly. Focus on what matters to you, your AI Co-Founder handles the rest.
We offer this whole set of features at a very competitive price. Stop juggling between many tools, Fluum is the all-in-one tool you need.

Following our rapid gain of users and traction, the codebase grows, the number of modules grows as well and the tools we need to use to monitor everything become important.

While we used to have build times around 4 to 5 minutes, it quickly became unpredictable, from 5 to 25 minutes, with some "Out of Memory" errors time to time. After a quick check, we found the culprit : Sentry.

Sentry is a tool that allows us to monitor our production frontend errors by sending us alerts whenever something bad happens. To run properly, it needs to gather sourcemaps from the NextJS build and upload them to Sentry, and that's where it starts eating up all the RAM

Initial setup

Following any documentation online, Sentry is said to be added at build time, in the next.config.js as follows :

const nextConfig = {
    ....
}

module.exports = withSentryConfig(nextConfig, {
   ...
})

Since this Sentry build happens inside Vercel, we are quickly limited by the resources offered and start to hit OOM errors

Error: Command "pnpm run build" exited with SIGKILL
▲ Build system report
▲ To always completely log this report, add VERCEL_BUILD_SYSTEM_REPORT=1 as an Environment Variable to your project.
• At least one "Out of Memory" ("OOM") event was detected during the build.
  • This occurs when processes or applications running during the build completely fill up the available memory (RAM) in the build container. When this happens, the build container terminates one of the processes during the build with a SIGKILL signal.
  • Read this troubleshooting guide for more information: https://vercel.link/troubleshoot-build-errors

Remediations

There are 2 solutions to this problem :

Remove Sentry and become blind
Remove Sentry... from NextJS build and handle it outside

We are of course going to opt for choice 2 :)

The key component here becomes GitHub Action : we are going to prebuild the NextJS project in GitHub Action, then run the Sentry CLI to create a new release with sourcemaps, and then deploy the prebuilt NextJS to Vercel using Vercel CLI

Now on Vercel, we unlink the GitHub repository and select the deployment mode from "NextJS" to "Others"

Here is the final Github Action file :

name: Vercel Production Deployment
env:
    VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
    VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID_PROD }}
    NODE_OPTIONS: '--max-old-space-size=4096' # 4 GB
on:
    push:
        branches:
            - master
jobs:
    notify-slack-starting-production-deployment:
        name: Notify Slack
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2

            - name: Get commit message
              id: commit_message
              run: |
                  # Escape quotes and newlines in commit message for JSON
                  COMMIT_MSG=$(git log -1 --pretty=%B | sed 's/"/\\"/g' | tr '\n' ' ' | sed 's/[[:space:]]*$//')
                  echo "::set-output name=message::$COMMIT_MSG"

            - name: Get commit hash
              id: commit_hash
              run: |
                  echo "::set-output name=hash::$(git rev-parse --short HEAD)"

            - name: Send Slack notification
              uses: slackapi/slack-github-action@v1.24.0
              with:
                  payload: |
                      {
                        "text": "Deploying Production Frontend | ${{ steps.commit_hash.outputs.hash }} : ${{ steps.commit_message.outputs.message }}"
                      }
              env:
                  SLACK_WEBHOOK_URL: ${{ secrets.SLACK_PROD_WEBHOOK_URL }}

    deploy:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - name: Install pnpm
              uses: pnpm/action-setup@v2
              with:
                  version: latest

            - name: Use Node.js
              uses: actions/setup-node@v3
              with:
                  node-version: '22'
                  cache: 'pnpm'

            - name: Cache pnpm store & node_modules
              uses: actions/cache@v3
              id: pnpm-cache
              with:
                  # Cache both the global store and node_modules
                  path: |
                      ~/.pnpm-store
                      node_modules
                  key: pnpm-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
                  restore-keys: |
                      pnpm-${{ runner.os }}-

            - name: Install Dependencies
              run: pnpm install --frozen-lockfile

            - name: Install Vercel CLI
              run: pnpm install --global vercel@latest

            - name: Pull Vercel Environment Information
              run: vercel pull --yes --environment=production --token=${{ secrets.VERCEL_TOKEN }}

            - name: Build Project Artifacts
              run: vercel build --debug --prod --token=${{ secrets.VERCEL_TOKEN }}

            - name: Install Sentry CLI
              run: pnpm install --global @sentry/cli@latest

            - name: Upload Sentry Source Maps
              # only on push to master
              if: github.event_name == 'push'
              run: |
                  npx @sentry/cli releases new $GITHUB_SHA
                  npx @sentry/cli releases files $GITHUB_SHA upload-sourcemaps .next --rewrite
                  npx @sentry/cli releases finalize $GITHUB_SHA
              env:
                  SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
                  SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
                  SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}

            - name: Deploy Project Artifacts to Vercel
              run: vercel deploy --prebuilt --prod --token=${{ secrets.VERCEL_TOKEN }}

Using this flow, our production builds are now back to approx. 5 minutes, and free of any Out of Memory error 🎉

Additional win : you can save on your organization costs by only having 1 seat in Vercel, now that this is the GitHub Action that is responsible for the deployment 💸

Hoping this solution will help many folks out there and save some time - keep growing 🚀

How I built an automated vulnerability scanner SECaaS

Ninjeneer — Tue, 06 Jun 2023 20:46:09 +0000

Few days ago, I released the Cyber Eye web platform. A SECaaS that allows any user, even those without technical knowledge, to run some basic & advanced security scans on their infrastructure.

Each of those scans take place from the outside, simulating what an attacker could be able to search for.

Today, we'll go down the technical side of this platform, and I'll explain how I did it. I am open to any suggestions to improve it :)

Server architecture

For this project, I decided to take as a priority the scalability. In fact, as users can themselves decide the periodicity of their scans (via a cron user-friendly selector), I expect some heavy loads during nights at 12AM.

So I decided to split the architecture this way :

A service managing the user requests responsible of validation, user credits checking, publishing the request to other services)
An AWS SQS queue in which the requests are sent
A service listening to the above queue, responsible of deploying the security probes (each probe being containerized with Docker)
An AWS SQS queue in which probes are sending notifications of results
A MongoDB atlas database storing the raw results of the probes
A service managing the probe results, to parse them and create a final report provided to the user
Finally, a real time database provided by Supabase to handle basic data storage and allow UI responsiveness
3 other additional services handling automated jobs, platform stats and billing using Stripe

The Request Service

This service is the entry point of the platform. It is responsible of handling user requests, making sure they have enough credits to run their scan and publishing the request to the AWS SQS Queue.

The Report Service

This service is the final piece of the workflow, triggered when the last probe of the scan is ended. It will pull all the probes results from Mongo, and then parse each of them based on a defined parser. Once every result is properly parsed, they are aggregated to form a final report embedding some more metadata. This report will be served to the user.

The Deployer Service

This one is the one that gave me headaches. Initially, I wanted to use an AWS Lambda function triggered by the AWS SQS Queue to deploy on demand some containers. However, I struggled a lot using ECS because it encourage the use of predefined configuration files, while I want my containers to be deployed on the fly without any structure.
At this point, my solutions were to dig deeper in AWS services hell or to go by myself and create my own very simple ECS. And that's what I did.

This homemade AWS ECS is written in Python, using the boto3 module to listen the AWS Queue. On each event received, the request is parsed, the relevant container is identified and a configuration is set. Then, the container is placed inside a deployment waiting queue (set randomly to 5 parallel run max.). When a deployment slot is free, the container is executed on the host machine, and killed at the end.

This choice also allows me to reduce a lot the costs that would have been generated by AWS.

Probes

Each probe is wrapped inside a Docker container to be able to run it almost anywhere, decoupled from the rest of the codebase, and written in any language
I basically do not create my own security tools, but build wrappers around famous tools like nmap, nikto, sqlmap... to provide trustful results.

Frontend architecture

Brace yourself, this gonna be a long read

I am using React and Tailwindcss :)

Git methology & CI / CD

To be able to manage easily my models and common code, I decided to go Monorepo for all the whole server side.
I set up some Github Actions to run my tests on each push/pull requests on my dev and production branches. Then, my docker images are built & stored using DockerHub.
On the CD side, I am working on setting up a watchtower on the production server to automatically pull new images and restart containers.

In conclusion, this project was for me an exciting playground where I could use all my knowledge and learnt a huge amount of new things (software architecture, DevOps practices, the rigor of production...). I will surely continue this project to bring it to a mature product, with more and more probes, and will post more updates of it on this blog

Thank you reading ! Happy to hear about your thoughts in the comment section

Creating a Netflix clone

Ninjeneer — Fri, 21 Jan 2022 16:27:08 +0000

Disclaimer

This project is for exercice purpose only. The whole source code will not be shared to avoid abuses.

Introduction

For the sake of the experimentation, I'm willing to build a Netflix clone, based on automated torrent downloading.

Requirements

To build a such system, I will need a web platform able to stream videos accross multiple devices as Netflix does. Hopefully, the Plex platform already does the job in a pretty awesome way. Therefore, I will just have to build a software able to search crawl in torrent websites in order to find and download the films and series I want to watch.

The idea is to have a web interface, asking me for the film I want to watch. If I already have it on my hard drive, it will open the Plex platform. If not, it will trigger an automated torrent download and move the film/serie into my plex media folder.

Technical stack

For this projet, I will use the Python language. As I haven't really worked with it yet, it will be a great introduction to this technology.

Steps

Step 1 : Getting the web page

After a research on the website, a specific URL is built. For instance, when searching for the avengers film, the URL looks like this : https://xxxxxxx/search/avengers/1/99/200

To process web scrapping, I am using the BeautifulSoup4 python module coupled with the requests one.

class XXXParser(Parser):
    def __init__(self):
        super().__init__()
        self.base_url = "https://xxx"

    def __build_url(self, film_name) -> str:
        url = self.base_url
        url += "/search/" + film_name + "/1/99/200"
        return url

    def __get_page_content(self, film_name: str) -> BeautifulSoup:
        html_content = requests.get(self.__build_url(film_name)).text
        return BeautifulSoup(html_content, 'html.parser')

The website I am scrapping is structured this way :

<tr>
    <td class="vertTh">
        <center>
            <a href="https://.../browse/200" title="More from this category">Video</a><br>
            (<a href="https://...y/browse/207" title="More from this category">HD - Movies</a>)
        </center>
    </td>
    <td>
        <div class="detName">
            <a href="https://.../torrent/34281763/Avengers.Endgame.2019.1080p.BRRip.x264-MP4"
                class="detLink">Avengers.Endgame.2019.1080p.BRRip.x264-MP4</a>
        </div>
        <a href="magnet:?...">
            <img src="https://.../static/img/icon-magnet.gif" alt="Magnet link" width="12"height="12">
        </a>

        <a href="https://.../user/..."><img src="https://.../static/img/trusted.png" alt="Trusted" title="Trusted" style="width:11px;" width="11" height="11" border="0"></a>
    </td>
    <td align="right">1803</td>
    <td align="right">383</td>
</tr>

After receiving the web page as a BeautifulSoup result object, I can start filtering the HTML tags to retrieve the information I am looking for :

Title
Download URL
Number of seeders
Trusted uploader
Video quality

page_content = self.__get_page_content(film_name)
rows = page_content.select("tr")
for row in rows:
    if row.select_one(".vertTh") is None:
      # This is not a table row containing a film
      continue

   film_name = row.select_one(".detLink").text
   film_url = row.select_one(".detLink").attrs.get('href')
   seeders = int(row.contents[len(row.contents) - 2].text)
   leechers = int(row.contents[len(row.contents) - 1].text)
   trusted = row.select_one("img[alt=Trusted]")
   quality = re.search("\\d{4}p", film_name)

Step 2 : filtering results

One of the problem with torrents name, is their unintelligable names. Many of them basically looks like this Avengers.Infinity.War.2018.1080p.10bit.BluRay.8CH.x265.HEVC-PSA which makes the work harder when it goes to filtering data.

So, I need to identify which text to remove to clear the titles.

Replace dots by spaces
Remove the quality using a regex (\d{3,4}p)
Remove the tags "DVDrip", "HDrip" etc... using a regex (\w{2,3}rip)
Remove repeted keywords among all titles : blueray, bluray, HEVC, AAC, ACC, PSA, MP4....
Remove encoding tags with regex ((x|h)\d+)
Remove useless "The" at the beggining of titles

I now have more natural results :

avengers
avengers endgame
avengers endgame (2019)
avengers infinity war
avengers infinity war 2018 english
avengers age of ultron (2015)

Step 3 : Sorting results

I don't want to spend time filtering the results myself to find the best one, I want it to be automated. That's why I need to give a score to each result based on several criteria.
By default, every result has a score of 0.

Levenshtein distance

This one is the most important of all scoring methods.

The levenshtein distance calculates the number of changes needed to go from a string A to a string B. In my case, I want the levenshtein distance to be the lower as possible between my query and the titles. Thanks to the previous title clearing done above, film titles already looks pretty natural.

Seeders

As I want my film to be downloaded as fast as possible, I'm looking for the ones with the most seeders. To avoid increasing too much the score based on the number of seeders, I am using the mathematical square root function, where the Y values increases slower as the X values increases.

Language

As a french speaker, I prefer watching french movies. If the movie title contains "french" keyword, then its score is increased by one. However, if it only contains a "fr" keyword, its score is increased by 0.5 because I am less sure it is a french language related tag.

Quality

The quality is also an important criteria. If the title contains a quality greater or equal than 1080p, the film's score increases of 1 points. If the quality is lower, it increases proportionnally to the quality (720p => 0.5, 480p => 0.25...)

Trusted uploader

The website I am scrapping has the ability to reward users with a tag "Trusted". This tag insures me a good quality and an accurate content. A film uploaded by a trusted uploader automatically increases its score by 1.

Step 4 : automate download

To be continued...

Thanks for reading, keep in mind to stay awesome !