Forem: Dmitriy Dmitriy

When an API stopped returning JSON, I switched to Selenium and added AI summaries

Dmitriy Dmitriy — Sat, 11 Apr 2026 02:24:55 +0000

I built a parser around the DNB Business Directory API. At first, everything worked fine — simple requests, JSON responses, clean and fast.

Then it suddenly stopped working.

My script started getting empty or unusable responses, even though the same requests still worked perfectly in the browser. Status codes were often 200, but the data was missing or incomplete.

After trying different headers, sessions, retries, and delays, it became clear that this wasn’t a normal API issue. Most likely, anti-bot filtering.

What I changed

Instead of trying to bypass it at the HTTP level, I switched to Selenium.

The new approach:

open the site in a real browser
search companies by keyword + country
paginate through results
collect company profile links
parse data directly from rendered pages

This worked immediately because it behaves like a real user session.

Then I added AI

After collecting company data, I wanted to understand what these companies actually do.

So I added a second stage:

scrape the company website (home + a few internal pages)
clean the text
send it to Groq
generate a short summary + list of services

I also added a simple keyword-based filter to detect risky content (gambling, adult, etc.) before sending data to the LLM.

Final pipeline

Selenium → company profiles → websites → multi-page scraping → AI summaries → Excel

Result

Interestingly, the workaround ended up being more useful than the original solution.

Instead of just collecting structured data, I now also get:

a quick description of each company
basic content classification
Demo / project

https://dibara512.github.io/my-site/

I built a company data pipeline — here’s what broke in real-world data

Dmitriy Dmitriy — Sat, 04 Apr 2026 17:55:37 +0000

I built a Python pipeline that automates company data collection:
searching registries, extracting websites, scraping content, finding phone numbers, and generating summaries using AI.

At first, I thought scraping would be the hard part.

It turned out to be the easiest.

What the pipeline does

The system:

finds companies via API (DNB)
extracts website and metadata
visits multiple pages of each site
extracts phone numbers from HTML, links and structured data
generates summaries using LLMs
exports everything into a structured Excel dataset

What actually broke

1. APIs are unstable and rate-limited

The DNB API would randomly return 429 and 503 errors.

Without retries, the pipeline would fail after a few requests.

I ended up implementing:

retry logic
exponential backoff
random delays between requests

Even then, stability is never guaranteed.

2. Phone numbers are surprisingly hard

Phone extraction turned out to be one of the hardest parts.

Numbers appear:

in different formats (+1, brackets, spaces, dashes)
mixed with dates or IDs
inside HTML text, links (tel:) and JSON-LD

I had to build logic to:

normalize formats
filter invalid matches
classify numbers by reliability

Without this step, the output was unusable.

3. Websites are inconsistent

Every site is different:

different HTML structures
missing data
broken markup
content hidden in scripts

There is no universal parser.

Even simple tasks like extracting clean text require handling multiple edge cases.

4. Matching company data is unreliable

Company names don’t always match exactly across sources.

Small differences (spacing, symbols, legal forms) break naive matching.

This forced me to implement stricter matching logic and fallbacks.

5. AI is helpful, but not deterministic

I used LLMs to generate company summaries from scraped text.

But even with the same input:

outputs vary
rate limits happen
some responses fail

To make it usable, I had to:

control prompts carefully
limit output length
add fallback between models

Results

The pipeline can process hundreds of companies per run,
replacing hours of manual work.

But the main takeaway:

Scraping is only a small part of the system.

Most of the effort goes into:

cleaning data
validating results
handling edge cases

Conclusion

If you’re starting with web scraping, the hardest part isn’t extracting data.

It’s making that data reliable.

Real-world data is messy, inconsistent and unpredictable.

Building a pipeline is not about scraping —
it’s about turning raw data into something usable.

Happy to share more details if there’s interest.

How I Automated Data Collection with Python (Web Scraping, AI, Excel)

Dmitriy Dmitriy — Tue, 24 Mar 2026 01:59:10 +0000

I recently built a set of Python tools that automate data collection, extract company information, and generate summaries using AI.

Instead of spending hours manually collecting data, everything is processed automatically.

Here’s a quick breakdown of how it works 👇

I’m a Python developer focused on automation, data extraction, and building practical tools for business workflows.

Instead of writing simple scripts, I build solutions that actually save time — collecting company data, automating repetitive processes, and integrating AI into real tasks.

If you want to see real examples of what I build, you can check my work here:

👉 https://dibara512.github.io/my-site/

What I build

Here are the main types of solutions I work on:

Web scraping tools (registries, directories, company databases)
Browser automation (forms, dashboards, repetitive workflows)
Excel and database processing tools
AI-powered data analysis (LLMs, summaries, classification)
Internal tools with GUI for teams

Real examples

1. Company data extraction from registries

I’ve built parsers for multiple national registries:

Finland
France (SIREN extraction)
Belgium, Austria, UK, Poland
Japan, Iceland

Typical workflow:

Read company list from Excel
Automatically search registry
Match exact company names
Extract registration numbers
Export results back to Excel

This allows processing hundreds of companies in minutes instead of hours.

2. Advanced contact scraper

One of my tools focuses on extracting contact data from websites:

Phone numbers (tel:, JSON-LD, raw HTML)
Company websites
Structured data

It includes filtering and validation, so the output is clean and ready to use.

3. AI-powered website analysis

I built a system that:

Loads multiple pages from a website
Extracts and aggregates content
Generates summaries using AI (Groq API)
Identifies business activity

This is especially useful when working with large datasets of unknown websites.

4. Full automation pipeline

One of the most advanced tools I developed combines everything:

Multi-page company search (up to 20 pages)
Matching results with Excel datasets
Collecting website, phone, industry, DUNS
AI-generated summaries
Structured export to Excel

This replaces hours of manual research and data entry.

You can find similar solutions here:

👉 https://dibara512.github.io/my-site/

Example: extracting frequent keywords from company data

Sometimes I need to analyze large datasets of company names or descriptions.

Here’s a simple example:

from collections import Counter
import re

def top_word_frequencies(text, min_len=3, top_n=20):
    text = text.lower()
    tokens = re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text)
    tokens = [t for t in tokens if len(t) >= min_len]
    counts = Counter(tokens)
    return counts.most_common(top_n)


text_data = """
Apple Inc Google LLC Microsoft Corporation Apple Google Apple
"""

top_words = top_word_frequencies(text_data)

for word, freq in top_words:
    print(word, freq)

This approach helps me:

identify patterns in company data
generate better keywords
improve search and matching logic Technologies I use

In most of my projects, I work with:

Python
Selenium / BeautifulSoup
Excel (openpyxl, pandas)
SQL (SQLite, Firebird)
APIs (Groq, OpenStreetMap)
Tkinter (for internal tools)
How I approach projects

My workflow is simple:

You describe the task
I propose a practical solution
I build a working tool
You get a ready-to-use result

No unnecessary complexity — only tools that solve the problem.

Final thoughts

Automation is not about writing scripts — it’s about saving time and reducing manual work.

If you're working with:

web scraping
automation
data collection
AI workflows

feel free to explore my work here:
👉 https://dibara512.github.io/my-site/