Forem: Pascal Clément

# How I built a fully automated Asian tech news pipeline with AI

Pascal Clément — Wed, 01 Apr 2026 08:13:56 +0000

Most Western developers have no idea what's happening in Asian tech. Not because nothing's happening — quite the opposite. The problem is the language barrier.

So I built AsiafeedTech: a fully automated pipeline that ingests, translates, scores, and turns Asian tech news into YouTube Shorts — without any human intervention.

Here's how it works:

The Pipeline

1. RSS Ingestion
25+ sources from China, Japan, Korea and SEA. Runs 3x daily via Spring Boot scheduler.

2. AI Translation & Filtering
Each article goes through Gemini with a carefully crafted prompt that translates, categorises, detects sponsored content, and filters non-tech articles in one call.

3. Viral Scoring
Gemini scores each article 1-10. We add category bonuses (AI +3, EV +2, Robotics +2) to surface the most relevant stories.

4. YouTube Shorts Generation
Top articles → Gemini script → Gemini TTS → Imagen 4.0 images → FFmpeg assembly → Whisper captions → YouTube upload.

The hardest part was getting consistent quality at every step. Each AI call needs precise prompting, output validation, and fallback logic.

Full writeup coming soon. Live at: https://asiafeedtech.com

We just upgraded the AsiaFeedTech content engine ⚡️

Pascal Clément — Mon, 30 Mar 2026 13:41:51 +0000

Here’s what’s new:

✅ Semantic deduplication (no more repeated news)
🎙️ Random AI voices (Kore / Charon)
🏷️ AI-generated tags
📺 Dynamic video length
🟥 ASS captions with highlight
🔔 Auto outro with subscribe

Cleaner. Smarter. More engaging.

visit us on https://www.asiafeedtech.com

Caption & Rendering Engine Upgrade

Pascal Clément — Sat, 28 Mar 2026 17:12:43 +0000

Update: Caption & Rendering Engine Upgrade

We improved the video pipeline with a focus on stability and readability:

Removed heavy zoom effects → lower memory usage
Optimized FFmpeg pipeline → fewer crashes (OOM fixes)
Introduced ASS-based captions → precise styling control
Improved contrast + font rendering for mobile

Result:
More stable rendering + significantly better readability.

new videos starts in: 8h

→ https://youtube.com/AsiaFeedTech/shorts

90% of Asian tech news never reaches the West.

Pascal Clément — Thu, 26 Mar 2026 08:42:37 +0000

90% of Asian tech news never reaches the West.
I built an AI Workflow that changes that:

📡 RSS from 25+ Asian sources
🤖 Translate → Script → Voice → Video → YouTube
Fully automated. Zero manual work.

👉 asiafeedtech.com
📺 youtube.com/@asiafeedtech

AI #AIWorkflow #TechNews #Asia #buildinpublic

# How we built a fully automated Asian tech news pipeline with AI

Pascal Clément — Mon, 23 Mar 2026 19:28:20 +0000

How we built a fully automated Asian tech news pipeline with AI

Most of the world's most interesting tech news originates in Asia — and most of it never makes it to English-speaking audiences.

We built AsiafeedTech to fix that.

What it does

Every 8 hours, our system:

Ingests 25+ RSS feeds from China, Japan, Korea and Southeast Asia
Translates and filters content using Gemini AI
Scores each article for viral potential (AI score + category weighting)
Surfaces only the best stories at asiafeedtech.com

The video pipeline

The top articles don't just become text posts — they become short-form videos, fully automated:

Gemini generates a YouTube script
Gemini TTS produces the voiceover
Pexels clips are sourced based on article context
FFmpeg assembles the 9:16 video
Whisper burns in captions
Branding overlay is applied
Upload to YouTube

The interesting engineering challenge: making a multi-step AI pipeline reliable, where quality compounds (or degrades) at each stage.

We'll be writing more about the architecture soon.

👉 https://asiafeedtech.com

ai #buildinpublic #webdev #opensource

AsiafeedTech – Asian tech news translated to English daily

Pascal Clément — Thu, 19 Mar 2026 11:54:16 +0000

I got tired of missing out on tech innovations coming out of Asia simply
because of the language barrier. So I built AsiafeedTech.

What it does

Aggregates 25+ RSS feeds from Chinese, Japanese and Korean tech publications
Translates articles daily using Google Gemini 2.5 Flash
Categorizes into AI, EV, Robotics, Startups, Hardware...

Tech Stack

Frontend: Angular 19 with SSR
Backend: Spring Boot 3.5 + PostgreSQL
AI: Gemini 2.5 Flash via REST API
Infrastructure: Docker

Interesting challenges

Parallel RSS ingestion with Spring @async (5 threads)
Prompt stored in DB – editable without redeployment
"Less like this" feedback updates the AI prompt automatically

Live: https://asiafeedtech.com

Kafka FinOps: How to Do Chargeback Reporting

Pascal Clément — Wed, 11 Mar 2026 06:55:18 +0000

If you run Kafka as shared infrastructure, you've probably faced this question at some point: who is responsible for this topic, and what does it cost us?

This is the core problem that Kafka FinOps tries to solve. In this post I'll explain what chargeback reporting means in a Kafka context, why it's hard, and how we implemented it in PartitionPilot.

What Is Chargeback Reporting?

Chargeback is a practice borrowed from cloud FinOps: instead of treating infrastructure costs as a single shared line item, you break them down by team, service, or product — and charge each one for what they actually use.

In AWS or GCP this is relatively straightforward. Cloud providers give you cost allocation tags. But Kafka has no native cost model. It doesn't know about teams, budgets, or ownership.

That's where chargeback reporting for Kafka comes in.

The Two Cost Drivers in Kafka

Before you can do chargeback, you need to understand what actually costs money in Kafka:

Storage — every message written to a topic is stored on disk until it expires (based on retention settings). A topic with a 7-day retention and high throughput can consume hundreds of gigabytes.

Traffic — every byte written to (bytes-in) and read from (bytes-out) a topic generates network traffic. On AWS MSK or Confluent Cloud, this traffic is billed directly.

Both can be measured via Prometheus JMX metrics:

kafka.log:type=Log,name=Size → storage per topic-partition
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec → inbound traffic per topic
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec → outbound traffic per topic

The Missing Piece: Ownership

Metrics alone aren't enough for chargeback. You also need to know who owns each topic.

In most Kafka deployments, topic ownership is tribal knowledge. It lives in someone's head, in a Confluence page that's three years out of date, or nowhere at all.

For chargeback to work, you need a system that:

Tracks which team or person owns each topic
Links cost metrics to that ownership
Produces a report that finance or engineering management can actually use

How PartitionPilot Implements This

PartitionPilot connects to your Prometheus endpoint and takes periodic cost snapshots. Each snapshot captures storage and traffic per topic, stamped with a timestamp.

On top of that, it lets you assign an owner to each topic and consumer group. Ownership is stored in a PostgreSQL database alongside the cost data.

The result: a chargeback report in CSV format that looks like this:

Owner       | Topic                  | Storage (GB) | Traffic In (GB) | Traffic Out (GB) | Estimated Cost
------------+------------------------+--------------+-----------------+------------------+---------------
Team A      | orders.v2              | 12.4         | 45.2            | 180.8            | CHF 23.40
Team B      | user-events            | 8.1          | 120.3           | 360.9            | CHF 41.20
Team C      | analytics.raw          | 95.2         | 890.1           | 2670.3           | CHF 312.80

This report can be exported and shared with engineering managers or finance teams on a monthly basis.

Why This Is Harder Than It Sounds

A few things make Kafka chargeback tricky in practice:

Topics are shared. A single topic can be written to by one team and consumed by three others. Who pays for the outbound traffic — the producer or the consumers? There's no universal answer. PartitionPilot lets you assign separate ownership for producer and consumer sides.

Retention makes storage non-obvious. The cost of a topic depends not just on throughput, but on retention settings. A low-traffic topic with 30-day retention can cost more than a high-traffic topic with 1-hour retention.

Metrics need aggregation. Raw Prometheus metrics are per-broker, per-partition. You need to aggregate them per topic across all brokers to get meaningful numbers.

Getting Started

PartitionPilot is self-hosted via Docker Compose. You can start a free 30-day trial at partitionpilot.com — no credit card required.

If your team is running Kafka as shared infrastructure and you want to start doing proper cost allocation, give it a try.

Pascal Clément — founder of PartitionPilot

How I Built Kafka Cost Tracking with Prometheus JMX

Pascal Clément — Wed, 11 Mar 2026 06:24:10 +0000

When you run Kafka at scale, you quickly realize that not all topics are created equal. Some topics consume gigabytes of storage. Others generate massive traffic. But without tooling, you have no idea which ones — or which team owns them.

This is the problem I ran into while building PartitionPilot, a Kafka cost management platform. In this post I'll explain how we track storage and traffic costs per topic using Prometheus JMX metrics.

The Problem: Kafka Has No Native Cost View

Kafka gives you offsets, consumer lag, and partition counts. What it doesn't give you is:

How much storage each topic is using
How much traffic (bytes in/out) each topic generates
Which team or service owns a topic
What that costs you per month

For small clusters this doesn't matter much. But once you have dozens of teams and hundreds of topics, the question "who is responsible for this 500GB topic?" becomes very real.

The Solution: Prometheus JMX Metrics

Kafka exposes JMX metrics that can be scraped by Prometheus. The two most useful metrics for cost tracking are:

kafka.log:type=Log,name=Size — the size in bytes of each topic-partition log on disk.

kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and BytesOutPerSec — the traffic rate per topic.

With these two metrics you can calculate:

Storage cost: bytes stored × your storage price per GB
Traffic cost: bytes transferred × your network price per GB

How PartitionPilot Uses These Metrics

PartitionPilot connects to your Prometheus endpoint (the one scraping Kafka JMX) and takes periodic snapshots. Each snapshot captures the current storage and traffic rates per topic.

Kafka JMX → Prometheus → PartitionPilot → Cost snapshot (PostgreSQL)

From these snapshots we calculate a rolling cost per topic, per day, per month. The result is a dashboard where you can see at a glance which topics are your "top talkers" — the ones driving most of your Kafka bill.

Assigning Ownership

Cost numbers alone aren't enough. You also need to know who owns each topic.

PartitionPilot lets you assign an owner (a person or team) to each topic and consumer group. Once ownership is assigned, you can generate a chargeback report: a CSV that breaks down Kafka costs by owner, ready to share with engineering managers or finance teams.

Exporting Metrics via /metrics

PartitionPilot also exposes a /metrics endpoint in Prometheus format. This means you can scrape PartitionPilot itself from your existing Prometheus setup and build Grafana dashboards on top of the cost and ownership data.

Works with Apache Kafka, AWS MSK, and Confluent

Any Kafka distribution that exposes JMX metrics via Prometheus is compatible. This includes:

Apache Kafka (self-hosted)
AWS MSK (Managed Streaming for Apache Kafka)
Confluent Platform

No agents or Kafka plugins are required — just a Prometheus scrape URL.

Getting Started

PartitionPilot is self-hosted via Docker Compose. You can start a free 30-day trial at partitionpilot.com — no credit card required.

If you're running Kafka and want to understand what it actually costs, give it a try. I'd love to hear your feedback.

Pascal Clément — founder of PartitionPilot