Forem: Vishwas Batra

The End of ScrapeForge

Vishwas Batra — Thu, 25 Dec 2025 03:00:29 +0000

This post was supposed to be:
Day 6 of building ScrapeForge 🛠️

But unfortunately I have to stop here.

I am genuinely sorry to everyone who followed this journey and supported the project. I can no longer continue building ScrapeForge at this time.

Why I am stopping?
The reason is simple and honest. Lack of funds.

I know this sounds like a common excuse but it is the truth.

Quick recap:
Yesterday I completed around 80% of Instagram profile scraper. After that I started researching deployment options.

Until now I had only deployed static or frontend focused apps and had no real experience deploying a production grade backend.

During this research I realized I missed an important detail during the planning phase on Day 2.

ScrapeForge is a complex backend. It is long running stateful and resource intensive.

This kind of system cannot run reliably on platforms like Netlify or Vercel. It needs proper cloud infrastructure.

After estimating the costs I realized running ScrapeForge on cloud infrastructure would cost around $150 to $220 per month. With a current budget of $0 this is simply not possible.

At this point I could ask for preorders or investments to fund the project. But that does not feel right to me.

So I have decided to pause ScrapeForge entirely. There will be no further dev logs after this one.

What is next?
For now I am focusing on freelancing to collect funds.

I will be building websites and projects and using the skills I gained while building ScrapeForge such as web scraping reverse engineering APIs and building crawlers.

Once I have enough funds I will return to ScrapeForge and continue full time.

If you want ScrapeForge to come back sooner please share this post with anyone who needs a developer or wants data scraped from websites. This would genuinely help me move faster.

Thank you to everyone who followed supported and believed in this journey.

Day 5 of building ScrapeForge 🛠️

Vishwas Batra — Wed, 24 Dec 2025 01:37:32 +0000

Day 5 was a big learning milestone.

While working on ScrapeForge, I explored backend APIs and learned how data can be accessed before it is rendered on the frontend. This made scraping significantly more straightforward at a technical level.

What surprised me was how many applications expose backend APIs without strong safeguards or request monitoring.

Key takeaway from today:
APIs must be treated as first class security surfaces and monitored continuously.

Day 6 will build on this.

Day 4 of building ScrapeForge 🛠️

Vishwas Batra — Tue, 23 Dec 2025 04:39:11 +0000

Day 4 was slow but informative.

I completed the remaining SERP scraping logic and then moved on to implementing proxy rotation. To understand the real cost of free solutions, I decided to test free proxies.

I manually collected around 2000 proxies and wrote a script to validate them. After nearly two hours of testing, only 139 were usable.

This exercise clearly showed why most production systems rely on paid proxy providers.
Lesson learned: free solutions often shift the cost from money to time.

Day 5 will continue from here.

Day 3 of building ScrapeForge 🛠️

Vishwas Batra — Mon, 22 Dec 2025 07:24:05 +0000

Day 3 was challenging but insightful.

After my usual routine of posting updates and solving one LeetCode problem, I started building the SERP scraper using Playwright.

Things worked at first, but I quickly encountered captchas, IP blacklisting, device fingerprinting, and multiple anti-bot layers.

I spent the rest of the day modifying the scripts to better mimic real users. This included adding delays, reusing browser contexts, and adjusting execution patterns.

I did not finish the scraper by the end of the day, but the learning was significant. It became very clear that web scraping is far more complex than it appears from the outside.

Full respect to anyone working in this space.
Day 4 will continue from here.

Day 2 building ScrapeForge 🛠️

Vishwas Batra — Sun, 21 Dec 2025 01:58:10 +0000

Building ScrapeForge in Public: Day 2

With the project brief done, Day 2 was about legality and foundations.

I spent the day researching the legal implications of this type of SaaS. The key conclusion was the need for a clear Terms of Service, to be drafted after backend development once features are fully defined.

After locking this decision, I initialized the repository and completed the standard project setup.

Key takeaway:
Before scaling code, make sure the legal boundaries are clearly defined.

Day 3 will focus on deeper execution.

Day 1 of building ScrapeForge in public

Vishwas Batra — Sat, 20 Dec 2025 02:18:27 +0000

Before writing any production code, I wanted clarity.

Day 1 was spent on three things:

Sharing the build in public intro
Solving one LeetCode problem to stay consistent with DSA
Creating a detailed project brief

The project brief helps me avoid missing important features, reduces the risk of rewrites, and keeps scope creep under control.

There is a quote I strongly believe in:
“Scope creep kills more projects than bad code ever will.”

By the end of the day, I had a complete plan covering build, launch, and marketing.

I am sharing the project brief template for other developers as well.
https://1drv.ms/w/c/7d1d62d19aca61de/IQAiVNKlZ0I-QoeZA-AX9EYTAYCEMD_VbCkn6OkHEN9wI-Q?e=RGG7aQ

On to Day 2. 🚀

Building ScrapeForge in public starting tomorrow 🚀

Vishwas Batra — Fri, 19 Dec 2025 06:13:53 +0000

Hi everyone 👋
I’m starting to build ScrapeForge in public tomorrow 🚀

ScrapeForge is an industrial-grade data scraping and analysis platform focused on reliable extraction, smart filtering, and AI-assisted insights.

I’ll be sharing progress updates, learnings, and behind-the-scenes decisions as the product takes shape.

Day 1 starts tomorrow. Looking forward to sharing the journey.