<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Pascal Clément</title>
    <description>The latest articles on Forem by Pascal Clément (@umbrincraft).</description>
    <link>https://forem.com/umbrincraft</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3817884%2Fb40e0ed9-f86c-4d8c-be2a-5db003d17be4.jpg</url>
      <title>Forem: Pascal Clément</title>
      <link>https://forem.com/umbrincraft</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/umbrincraft"/>
    <language>en</language>
    <item>
      <title>How I built a 24/7 YouTube livestreaming SaaS with Spring Boot, FFmpeg &amp; Imagen</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Mon, 27 Apr 2026 13:51:36 +0000</pubDate>
      <link>https://forem.com/umbrincraft/how-i-built-a-247-youtube-livestreaming-saas-with-spring-boot-ffmpeg-imagen-fa6</link>
      <guid>https://forem.com/umbrincraft/how-i-built-a-247-youtube-livestreaming-saas-with-spring-boot-ffmpeg-imagen-fa6</guid>
      <description>&lt;h1&gt;
  
  
  How I built a 24/7 YouTube livestreaming SaaS with Spring Boot, FFmpeg &amp;amp; Imagen
&lt;/h1&gt;

&lt;p&gt;Running a 24/7 YouTube stream sounds simple until you actually try it. OBS crashes. The machine sleeps. The stream drops.&lt;/p&gt;

&lt;p&gt;I built IteraMachina to solve this — a SaaS platform where you compose scenes, build playlists, and stream to YouTube Live forever, fully managed in the cloud.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Architecture
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Angular SSR&lt;/strong&gt; frontend — Scenes, Compositions, Runner in a 3-column layout&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spring Boot 3.x&lt;/strong&gt; backend — SceneService, CompositionService, RenderService, StreamService&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Imagen AI&lt;/strong&gt; — generates visuals per scene from a text prompt&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FFmpeg&lt;/strong&gt; — assembles video from scenes + audio, streams via RTMP to &lt;code&gt;rtmp://a.rtmp.youtube.com/live2&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PostgreSQL&lt;/strong&gt; — scenes, compositions, streams&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stripe&lt;/strong&gt; — Basic / Pro subscription plans&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Docker&lt;/strong&gt; on Linode&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Workflow
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Create scenes (image prompt + audio track)&lt;/li&gt;
&lt;li&gt;Arrange scenes into a composition&lt;/li&gt;
&lt;li&gt;Add compositions to a playlist (drag &amp;amp; drop reorder)&lt;/li&gt;
&lt;li&gt;Hit Run — the stream starts and loops indefinitely&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The trickiest part: &lt;a class="mentioned-user" href="https://dev.to/async"&gt;@async&lt;/a&gt; + @Transactional
&lt;/h2&gt;

&lt;p&gt;The render pipeline is async — FFmpeg can take minutes. The trap: Spring's &lt;code&gt;@Async&lt;/code&gt; and &lt;code&gt;@Transactional&lt;/code&gt; can't live on the same method or the transaction context doesn't propagate correctly.&lt;/p&gt;

&lt;p&gt;Solution: split into three distinct methods — &lt;code&gt;loadScenes()&lt;/code&gt; (@Transactional readOnly), &lt;code&gt;setStatus()&lt;/code&gt; (@Transactional), &lt;code&gt;renderAsync()&lt;/code&gt; (&lt;a class="mentioned-user" href="https://dev.to/async"&gt;@async&lt;/a&gt; only). Took longer to debug than I'd like to admit.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;Plan limits enforcement (Basic: 1 project, 3 scenes), Stripe customer portal, and community scene sharing.&lt;/p&gt;

&lt;p&gt;Live at 👉 iteramachina.com — built under the Umbrincraft umbrella.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What would you add to a streaming SaaS like this? Curious what use cases others see.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>springboot</category>
      <category>ffmpeg</category>
      <category>saas</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>How I use Gemini to score news virality — and why it actually works</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Mon, 27 Apr 2026 13:17:47 +0000</pubDate>
      <link>https://forem.com/umbrincraft/how-i-use-gemini-to-score-news-virality-and-why-it-actually-works-1aad</link>
      <guid>https://forem.com/umbrincraft/how-i-use-gemini-to-score-news-virality-and-why-it-actually-works-1aad</guid>
      <description>&lt;h1&gt;
  
  
  How I use Gemini to score news virality — and why it actually works
&lt;/h1&gt;

&lt;p&gt;When I built AsiafeedTech, I had a problem: 200+ Asian tech articles per day, but only 5 slots in the newsletter and 3 YouTube Shorts to produce.&lt;/p&gt;

&lt;p&gt;Something had to decide which stories mattered. I built a viral scoring system powered by Gemini — and it turned out to be one of the most interesting engineering challenges of the project.&lt;/p&gt;

&lt;h2&gt;
  
  
  The naive approach (and why it fails)
&lt;/h2&gt;

&lt;p&gt;My first instinct was keyword matching. If an article mentions "OpenAI" or "BYD" or "IPO" — high score. Simple, fast, wrong.&lt;/p&gt;

&lt;p&gt;The problem: keyword matching finds popular topics, not viral stories. An article titled &lt;em&gt;"BYD reports quarterly results"&lt;/em&gt; and &lt;em&gt;"BYD just quietly shipped a car that charges in 5 minutes"&lt;/em&gt; both match "BYD" — but only one of them spreads.&lt;/p&gt;

&lt;h2&gt;
  
  
  What makes something actually go viral?
&lt;/h2&gt;

&lt;p&gt;I broke it down into signals that a human editor would intuitively feel:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Novelty&lt;/strong&gt; — is this genuinely new information, or a rehash?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Surprise factor&lt;/strong&gt; — does it contradict expectations?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stakes&lt;/strong&gt; — does it affect a lot of people or a lot of money?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Specificity&lt;/strong&gt; — concrete numbers beat vague claims every time&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Western relevance&lt;/strong&gt; — will a founder in SF actually care?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last one is critical for AsiafeedTech. A story about a Chinese domestic policy change might be huge locally but irrelevant to my audience. Gemini needs to apply a cultural filter, not just a relevance filter.&lt;/p&gt;

&lt;h2&gt;
  
  
  The prompt
&lt;/h2&gt;

&lt;p&gt;I pass the translated article title + summary to Gemini with a structured scoring prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You are an editor for a Western tech audience (founders, investors, developers).
Score this Asian tech news article on a scale of 1-100 for viral potential.

Consider:
- Novelty and surprise factor
- Relevance to Western tech/startup ecosystem  
- Specificity (concrete numbers, named companies, tangible outcomes)
- Stakes (market size, geopolitical impact, technological leap)
- Emotional resonance (inspiring, alarming, surprising)

Return ONLY a JSON object:
{"score": 85, "reason": "one sentence"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;reason&lt;/code&gt; field is gold — it's what I display in the admin panel and what Gemini uses as a forcing function to justify the score rather than guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  Temperature matters more than you think
&lt;/h2&gt;

&lt;p&gt;I run scoring at &lt;code&gt;temperature: 0.2&lt;/code&gt;. Low temperature = consistent, repeatable scores across similar articles. Higher temperature introduced too much variance — the same article would score 45 one run and 72 the next.&lt;/p&gt;

&lt;p&gt;For creative tasks like script generation I use &lt;code&gt;temperature: 0.9&lt;/code&gt;. For scoring: determinism wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does it actually work?
&lt;/h2&gt;

&lt;p&gt;Honestly — better than I expected. The top-scored articles are consistently the ones I'd have picked manually. Stories about Chinese EV range breakthroughs, surprise AI model releases, or unexpected regulatory moves consistently outscore routine earnings reports.&lt;/p&gt;

&lt;p&gt;The system isn't perfect. It occasionally over-scores geopolitical news that's more alarming than actionable, and under-scores slow-burn trends. But for a fully automated pipeline with zero human editors — it holds up.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;I'm considering adding a feedback loop: if a YouTube Short generated from a high-scored article gets above-average views, that reinforces the scoring pattern. Essentially fine-tuning the prompt based on actual engagement data.&lt;/p&gt;

&lt;p&gt;Building in public at 👉 asiafeedtech.com&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What scoring signals would you add? Drop them in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>gemini</category>
      <category>ai</category>
      <category>buildinpublic</category>
      <category>programming</category>
    </item>
    <item>
      <title>How I Built a Fully Automated Asian Tech News Newsletter with Spring Boot + Gemini AI</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Sun, 19 Apr 2026 10:33:14 +0000</pubDate>
      <link>https://forem.com/umbrincraft/how-i-built-a-fully-automated-asian-tech-news-newsletter-with-spring-boot-gemini-ai-15m3</link>
      <guid>https://forem.com/umbrincraft/how-i-built-a-fully-automated-asian-tech-news-newsletter-with-spring-boot-gemini-ai-15m3</guid>
      <description>&lt;h1&gt;
  
  
  How I Built a Fully Automated Asian Tech News Newsletter with Spring Boot + Gemini AI
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;From RSS ingestion to your inbox — zero manual work.&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;p&gt;Asian tech markets move fast. Chinese AI startups, semiconductor breakthroughs, EV innovations — most of it never makes it to Western tech media in real time. And when it does, it's already old news.&lt;/p&gt;

&lt;p&gt;I wanted a daily digest of the top Asian tech stories, translated into English, delivered automatically. So I built it.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Stack
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Spring Boot 3&lt;/strong&gt; — backend pipeline&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PostgreSQL + Flyway&lt;/strong&gt; — storage and migrations&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Gemini 2.5 Flash&lt;/strong&gt; — translation, scoring, and newsletter writing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Buttondown&lt;/strong&gt; — newsletter delivery (free plan, full API access)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Angular SSR&lt;/strong&gt; — frontend&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Docker Compose&lt;/strong&gt; on Linode — production deployment&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Pipeline
&lt;/h2&gt;

&lt;p&gt;The whole system runs automatically, twice a day at 10:00 and 17:00 Shanghai time.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1 — RSS Ingestion
&lt;/h3&gt;

&lt;p&gt;A Spring Boot scheduler fetches RSS feeds from ~15 Chinese and Asian tech sources (ithome, huxiu, 36kr, etc.). Each article is deduplicated by URL and stored in PostgreSQL.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Scheduled&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cron&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"${app.run-morning-cron}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;zone&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"Asia/Shanghai"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;morningRun&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2 — Translation + Viral Scoring
&lt;/h3&gt;

&lt;p&gt;Each new article gets sent to Gemini for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Translation to English (title + description)&lt;/li&gt;
&lt;li&gt;A viral score (0–100) based on category, topic, and potential Western interest
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="no"&gt;SYSTEM_PROMPT_SCORING&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"""
    You are an Asian tech news editor for a Western audience.
    Score each article from 0-100 based on viral potential.
    Consider: AI, semiconductors, EVs, robotics = high score.
    Local business news = low score.
    Return ONLY a number.
    """&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Category bonuses are applied on top — AI stories get +15, EV stories get +10, etc.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3 — Newsletter Generation
&lt;/h3&gt;

&lt;p&gt;A separate &lt;code&gt;buttondown-generator&lt;/code&gt; service (standalone Spring Boot app) runs on its own schedule:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Calls &lt;code&gt;GET /api/feeds/buttondown?limit=5&lt;/code&gt; on the main backend — returns the top 5 unprocessed stories&lt;/li&gt;
&lt;li&gt;Sends them to Gemini to write a newsletter in HTML format&lt;/li&gt;
&lt;li&gt;Posts the result to Buttondown via API&lt;/li&gt;
&lt;li&gt;Writes the Buttondown email ID back to each article in the DB
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;ObjectNode&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;objectMapper&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;createObjectNode&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"subject"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newsletter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getTitle&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"body"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newsletter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getContentHtml&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"status"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"about_to_send"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

&lt;span class="n"&gt;restClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;post&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;uri&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"/emails"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;header&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Authorization"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"Token "&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;apiKey&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;header&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"X-Buttondown-Live-Dangerously"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;contentType&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;MediaType&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;APPLICATION_JSON&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;retrieve&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 4 — The Flag System
&lt;/h3&gt;

&lt;p&gt;Each article in the DB has flags per generator:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;aft_feed&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;buttondown_used&lt;/span&gt; &lt;span class="nb"&gt;BOOLEAN&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="k"&gt;FALSE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;aft_feed&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;youtube_used&lt;/span&gt; &lt;span class="nb"&gt;BOOLEAN&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="k"&gt;FALSE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This ensures no article is sent twice to the same channel, and each generator is fully independent. Adding a new generator (TikTok, LinkedIn, etc.) is just a new flag + a new endpoint.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Architecture Decision — Separate Generator Services
&lt;/h2&gt;

&lt;p&gt;Each generator (YouTube, Buttondown) is a &lt;strong&gt;separate Spring Boot app&lt;/strong&gt; with its own Docker container, its own scheduler, and its own port.&lt;/p&gt;

&lt;p&gt;Why not one monolith?&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Independent deployment — update the newsletter generator without touching the YouTube pipeline&lt;/li&gt;
&lt;li&gt;Independent failure — if Buttondown has an outage, YouTube keeps running&lt;/li&gt;
&lt;li&gt;Independent scaling — the YouTube generator is CPU-heavy (FFmpeg), the newsletter generator is not&lt;/li&gt;
&lt;li&gt;Clean separation of concerns&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The main &lt;code&gt;asiafeedtech-backend&lt;/code&gt; only exposes REST endpoints. Generators are clients.&lt;/p&gt;




&lt;h2&gt;
  
  
  What It Looks Like
&lt;/h2&gt;

&lt;p&gt;Every morning and evening, this lands in subscribers' inboxes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AsiafeedTech Daily — 2026-04-19&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Yuanjie Technology: From 'Scammer' to A-share King&lt;/strong&gt;&lt;br&gt;
Yuanjie Technology has become China's highest-priced A-share stock, soaring 12x in a year...&lt;br&gt;
&lt;em&gt;This signals a shift in China's investment focus from consumer goods to high-tech hardware.&lt;/em&gt;&lt;br&gt;
&lt;a href="https://asiafeedtech.com" rel="noopener noreferrer"&gt;Read more →&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Lessons Learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Buttondown's "API access" on free plan is misleading.&lt;/strong&gt; The general API is free, but the &lt;code&gt;POST /emails&lt;/code&gt; endpoint (creating and sending posts) requires the Enterprise plan — unless you use the &lt;code&gt;X-Buttondown-Live-Dangerously&lt;/code&gt; header, which bypasses the confirmation requirement for &lt;code&gt;about_to_send&lt;/code&gt; status. This took a few hours to figure out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Gemini rate limiting is real.&lt;/strong&gt; For bulk scoring, we use a &lt;code&gt;Semaphore(2)&lt;/code&gt; + delays + 429 detection. Flash is fast but not infinitely parallel.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Separate the &lt;code&gt;@Async&lt;/code&gt; and &lt;code&gt;@Transactional&lt;/code&gt; concerns in Spring Boot 3.&lt;/strong&gt; Putting both on the same method causes proxy issues. Split into three methods: one for loading data (&lt;code&gt;@Transactional(readOnly=true)&lt;/code&gt;), one for status updates (&lt;code&gt;@Transactional&lt;/code&gt;), one for async work (&lt;code&gt;@Async&lt;/code&gt; only).&lt;/p&gt;




&lt;h2&gt;
  
  
  What's Next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;TikTok generator&lt;/strong&gt; — same pipeline, different output format&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paid API access&lt;/strong&gt; — expose the scored feed as a paid REST API for media companies and investors&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;More sources&lt;/strong&gt; — Japan, Korea, Southeast Asia&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Subscribe
&lt;/h2&gt;

&lt;p&gt;If you want daily Asian tech news in your inbox:&lt;br&gt;
👉 &lt;a href="https://buttondown.com/asiafeedtech" rel="noopener noreferrer"&gt;buttondown.com/asiafeedtech&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Free. No spam. Unsubscribe anytime.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Built with Spring Boot, Gemini AI, Buttondown, and too much coffee.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>java</category>
      <category>springboot</category>
      <category>showdev</category>
      <category>newsletter</category>
    </item>
    <item>
      <title>AsiafeedTech — April Update: AI Image Gallery &amp; Video Engine Optimization</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Fri, 17 Apr 2026 06:27:13 +0000</pubDate>
      <link>https://forem.com/umbrincraft/asiafeedtech-april-update-ai-image-gallery-video-engine-optimization-knh</link>
      <guid>https://forem.com/umbrincraft/asiafeedtech-april-update-ai-image-gallery-video-engine-optimization-knh</guid>
      <description>&lt;h2&gt;
  
  
  AsiafeedTech — April Update: AI Image Gallery &amp;amp; Video Engine Optimization
&lt;/h2&gt;

&lt;p&gt;A quick update on what shipped this month in the AsiafeedTech pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  AI Visual Gallery
&lt;/h3&gt;

&lt;p&gt;Every article now gets a unique cover image generated by Imagen 4.0. The prompt is built from the article title + category context + a fixed cinematic style directive. Images are saved to persistent storage and served via a proxy endpoint in the main backend — the browser never talks to the generator service directly.&lt;/p&gt;

&lt;p&gt;You can browse the gallery at &lt;strong&gt;asiafeedtech.com/gallery&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Video Engine Optimization
&lt;/h3&gt;

&lt;p&gt;The Shorts pipeline went through several rounds of improvement:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;aformat&lt;/code&gt; normalization per stream for consistent audio sync in complex FFmpeg filter graphs&lt;/li&gt;
&lt;li&gt;Dynamic font sizing for title cards based on character count&lt;/li&gt;
&lt;li&gt;Whisper caption overflow fix via &lt;code&gt;WrapStyle&lt;/code&gt; in the ASS template&lt;/li&gt;
&lt;li&gt;Improved Imagen prompt structure — category-specific visual concepts per news category&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  TikTok Integration
&lt;/h3&gt;

&lt;p&gt;The TikTok Content Posting API app is currently under review. Cross-posting will be automatic once approved — same flow as YouTube upload.&lt;/p&gt;

&lt;h3&gt;
  
  
  Stack
&lt;/h3&gt;

&lt;p&gt;Angular SSR · Spring Boot · PostgreSQL · Flyway · Gemini · Imagen 4.0 · FFmpeg · Whisper · Docker · Linode&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://asiafeedtech.com" rel="noopener noreferrer"&gt;asiafeedtech.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>productivity</category>
      <category>asiafeedtech</category>
    </item>
    <item>
      <title>## AsiafeedTech — April Update: AI Image Gallery &amp; Video Engine Optimization

A quick update on what shipped this month in the AsiafeedTech pipeline.

### AI Visual Gallery

Every article now gets a unique cover image generated by Imagen 4.0. The prompt is</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Fri, 17 Apr 2026 06:24:33 +0000</pubDate>
      <link>https://forem.com/umbrincraft/-asiafeedtech-april-update-ai-image-gallery-video-engine-optimization-a-quick-update-on-121g</link>
      <guid>https://forem.com/umbrincraft/-asiafeedtech-april-update-ai-image-gallery-video-engine-optimization-a-quick-update-on-121g</guid>
      <description></description>
    </item>
    <item>
      <title># How I built a fully automated Asian tech news pipeline with AI</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Wed, 01 Apr 2026 08:13:56 +0000</pubDate>
      <link>https://forem.com/umbrincraft/-how-i-built-a-fully-automated-asian-tech-news-pipeline-with-ai-4oe3</link>
      <guid>https://forem.com/umbrincraft/-how-i-built-a-fully-automated-asian-tech-news-pipeline-with-ai-4oe3</guid>
      <description>&lt;p&gt;Most Western developers have no idea what's happening in Asian tech. Not because nothing's happening — quite the opposite. The problem is the language barrier.&lt;/p&gt;

&lt;p&gt;So I built AsiafeedTech: a fully automated pipeline that ingests, translates, scores, and turns Asian tech news into YouTube Shorts — without any human intervention.&lt;/p&gt;

&lt;p&gt;Here's how it works:&lt;/p&gt;

&lt;h2&gt;
  
  
  The Pipeline
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. RSS Ingestion&lt;/strong&gt;&lt;br&gt;
25+ sources from China, Japan, Korea and SEA. Runs 3x daily via Spring Boot scheduler.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. AI Translation &amp;amp; Filtering&lt;/strong&gt;&lt;br&gt;
Each article goes through Gemini with a carefully crafted prompt that translates, categorises, detects sponsored content, and filters non-tech articles in one call.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Viral Scoring&lt;/strong&gt;&lt;br&gt;
Gemini scores each article 1-10. We add category bonuses (AI +3, EV +2, Robotics +2) to surface the most relevant stories.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. YouTube Shorts Generation&lt;/strong&gt;&lt;br&gt;
Top articles → Gemini script → Gemini TTS → Imagen 4.0 images → FFmpeg assembly → Whisper captions → YouTube upload.&lt;/p&gt;

&lt;p&gt;The hardest part was getting consistent quality at every step. Each AI call needs precise prompting, output validation, and fallback logic.&lt;/p&gt;

&lt;p&gt;Full writeup coming soon. Live at: &lt;a href="https://asiafeedtech.com" rel="noopener noreferrer"&gt;https://asiafeedtech.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>workflow</category>
      <category>automated</category>
    </item>
    <item>
      <title>We just upgraded the AsiaFeedTech content engine ⚡️</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Mon, 30 Mar 2026 13:41:51 +0000</pubDate>
      <link>https://forem.com/umbrincraft/we-just-upgraded-the-asiafeedtech-content-engine-3j96</link>
      <guid>https://forem.com/umbrincraft/we-just-upgraded-the-asiafeedtech-content-engine-3j96</guid>
      <description>&lt;p&gt;Here’s what’s new:&lt;/p&gt;

&lt;p&gt;✅ Semantic deduplication (no more repeated news)&lt;br&gt;
🎙️ Random AI voices (Kore / Charon)&lt;br&gt;
🏷️ AI-generated tags&lt;br&gt;
📺 Dynamic video length&lt;br&gt;
🟥 ASS captions with highlight&lt;br&gt;
🔔 Auto outro with subscribe&lt;/p&gt;

&lt;p&gt;Cleaner. Smarter. More engaging.&lt;/p&gt;

&lt;p&gt;visit us on &lt;a href="https://www.asiafeedtech.com" rel="noopener noreferrer"&gt;https://www.asiafeedtech.com&lt;/a&gt; &lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0a1eolvgp0qyhp8n2biq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0a1eolvgp0qyhp8n2biq.png" alt=" " width="800" height="457"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>contentcreation</category>
      <category>tech</category>
    </item>
    <item>
      <title>Caption &amp; Rendering Engine Upgrade</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Sat, 28 Mar 2026 17:12:43 +0000</pubDate>
      <link>https://forem.com/umbrincraft/caption-rendering-engine-upgrade-3aah</link>
      <guid>https://forem.com/umbrincraft/caption-rendering-engine-upgrade-3aah</guid>
      <description>&lt;p&gt;Update: Caption &amp;amp; Rendering Engine Upgrade&lt;/p&gt;

&lt;p&gt;We improved the video pipeline with a focus on stability and readability:&lt;/p&gt;

&lt;p&gt;Removed heavy zoom effects → lower memory usage&lt;br&gt;
Optimized FFmpeg pipeline → fewer crashes (OOM fixes)&lt;br&gt;
Introduced ASS-based captions → precise styling control&lt;br&gt;
Improved contrast + font rendering for mobile&lt;/p&gt;

&lt;p&gt;Result:&lt;br&gt;
More stable rendering + significantly better readability.&lt;/p&gt;

&lt;p&gt;new videos starts in: 8h&lt;/p&gt;

&lt;p&gt;→ &lt;a href="https://youtube.com/AsiaFeedTech/shorts" rel="noopener noreferrer"&gt;https://youtube.com/AsiaFeedTech/shorts&lt;/a&gt;&lt;/p&gt;

</description>
      <category>devlog</category>
      <category>discord</category>
      <category>ai</category>
      <category>workflow</category>
    </item>
    <item>
      <title>90% of Asian tech news never reaches the West.</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Thu, 26 Mar 2026 08:42:37 +0000</pubDate>
      <link>https://forem.com/umbrincraft/90-of-asian-tech-news-never-reaches-the-west-3k1f</link>
      <guid>https://forem.com/umbrincraft/90-of-asian-tech-news-never-reaches-the-west-3k1f</guid>
      <description>&lt;p&gt;90% of Asian tech news never reaches the West.&lt;br&gt;
I built an AI Workflow that changes that:&lt;/p&gt;

&lt;p&gt;📡 RSS from 25+ Asian sources&lt;br&gt;
🤖 Translate → Script → Voice → Video → YouTube&lt;br&gt;
Fully automated. Zero manual work.&lt;/p&gt;

&lt;p&gt;👉 asiafeedtech.com&lt;br&gt;
📺 youtube.com/@asiafeedtech&lt;/p&gt;

&lt;h1&gt;
  
  
  AI #AIWorkflow #TechNews #Asia #buildinpublic
&lt;/h1&gt;

</description>
    </item>
    <item>
      <title># How we built a fully automated Asian tech news pipeline with AI</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Mon, 23 Mar 2026 19:28:20 +0000</pubDate>
      <link>https://forem.com/umbrincraft/-how-we-built-a-fully-automated-asian-tech-news-pipeline-with-ai-3ajd</link>
      <guid>https://forem.com/umbrincraft/-how-we-built-a-fully-automated-asian-tech-news-pipeline-with-ai-3ajd</guid>
      <description>&lt;h1&gt;
  
  
  How we built a fully automated Asian tech news pipeline with AI
&lt;/h1&gt;

&lt;p&gt;Most of the world's most interesting tech news originates in Asia — and most of it never makes it to English-speaking audiences.&lt;/p&gt;

&lt;p&gt;We built AsiafeedTech to fix that.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;p&gt;Every 8 hours, our system:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ingests 25+ RSS feeds from China, Japan, Korea and Southeast Asia&lt;/li&gt;
&lt;li&gt;Translates and filters content using Gemini AI&lt;/li&gt;
&lt;li&gt;Scores each article for viral potential (AI score + category weighting)&lt;/li&gt;
&lt;li&gt;Surfaces only the best stories at asiafeedtech.com&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The video pipeline
&lt;/h2&gt;

&lt;p&gt;The top articles don't just become text posts — they become short-form videos, fully automated:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Gemini generates a YouTube script&lt;/li&gt;
&lt;li&gt;Gemini TTS produces the voiceover&lt;/li&gt;
&lt;li&gt;Pexels clips are sourced based on article context&lt;/li&gt;
&lt;li&gt;FFmpeg assembles the 9:16 video&lt;/li&gt;
&lt;li&gt;Whisper burns in captions&lt;/li&gt;
&lt;li&gt;Branding overlay is applied&lt;/li&gt;
&lt;li&gt;Upload to YouTube&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The interesting engineering challenge: making a multi-step AI pipeline reliable, where quality compounds (or degrades) at each stage.&lt;/p&gt;

&lt;p&gt;We'll be writing more about the architecture soon.&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://asiafeedtech.com" rel="noopener noreferrer"&gt;https://asiafeedtech.com&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  ai #buildinpublic #webdev #opensource
&lt;/h1&gt;

</description>
      <category>asiafeedtech</category>
      <category>ai</category>
      <category>workflow</category>
    </item>
    <item>
      <title>AsiafeedTech – Asian tech news translated to English daily</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Thu, 19 Mar 2026 11:54:16 +0000</pubDate>
      <link>https://forem.com/umbrincraft/asiafeedtech-asian-tech-news-translated-to-english-daily-1b74</link>
      <guid>https://forem.com/umbrincraft/asiafeedtech-asian-tech-news-translated-to-english-daily-1b74</guid>
      <description>&lt;p&gt;I got tired of missing out on tech innovations coming out of Asia simply &lt;br&gt;
because of the language barrier. So I built AsiafeedTech.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Aggregates 25+ RSS feeds from Chinese, Japanese and Korean tech publications&lt;/li&gt;
&lt;li&gt;Translates articles daily using Google Gemini 2.5 Flash&lt;/li&gt;
&lt;li&gt;Categorizes into AI, EV, Robotics, Startups, Hardware...&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Tech Stack
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frontend:&lt;/strong&gt; Angular 19 with SSR&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend:&lt;/strong&gt; Spring Boot 3.5 + PostgreSQL&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI:&lt;/strong&gt; Gemini 2.5 Flash via REST API&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Infrastructure:&lt;/strong&gt; Docker&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interesting challenges
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Parallel RSS ingestion with Spring &lt;a class="mentioned-user" href="https://dev.to/async"&gt;@async&lt;/a&gt; (5 threads)&lt;/li&gt;
&lt;li&gt;Prompt stored in DB – editable without redeployment&lt;/li&gt;
&lt;li&gt;"Less like this" feedback updates the AI prompt automatically&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Live: &lt;a href="https://asiafeedtech.com" rel="noopener noreferrer"&gt;https://asiafeedtech.com&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Kafka FinOps: How to Do Chargeback Reporting</title>
      <dc:creator>Pascal Clément</dc:creator>
      <pubDate>Wed, 11 Mar 2026 06:55:18 +0000</pubDate>
      <link>https://forem.com/umbrincraft/kafka-finops-how-to-do-chargeback-reporting-8g8</link>
      <guid>https://forem.com/umbrincraft/kafka-finops-how-to-do-chargeback-reporting-8g8</guid>
      <description>&lt;p&gt;If you run Kafka as shared infrastructure, you've probably faced this question at some point: &lt;strong&gt;who is responsible for this topic, and what does it cost us?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is the core problem that Kafka FinOps tries to solve. In this post I'll explain what chargeback reporting means in a Kafka context, why it's hard, and how we implemented it in &lt;a href="https://partitionpilot.com" rel="noopener noreferrer"&gt;PartitionPilot&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is Chargeback Reporting?
&lt;/h2&gt;

&lt;p&gt;Chargeback is a practice borrowed from cloud FinOps: instead of treating infrastructure costs as a single shared line item, you break them down by team, service, or product — and charge each one for what they actually use.&lt;/p&gt;

&lt;p&gt;In AWS or GCP this is relatively straightforward. Cloud providers give you cost allocation tags. But Kafka has no native cost model. It doesn't know about teams, budgets, or ownership.&lt;/p&gt;

&lt;p&gt;That's where chargeback reporting for Kafka comes in.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Two Cost Drivers in Kafka
&lt;/h2&gt;

&lt;p&gt;Before you can do chargeback, you need to understand what actually costs money in Kafka:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Storage&lt;/strong&gt; — every message written to a topic is stored on disk until it expires (based on retention settings). A topic with a 7-day retention and high throughput can consume hundreds of gigabytes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Traffic&lt;/strong&gt; — every byte written to (bytes-in) and read from (bytes-out) a topic generates network traffic. On AWS MSK or Confluent Cloud, this traffic is billed directly.&lt;/p&gt;

&lt;p&gt;Both can be measured via Prometheus JMX metrics:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;kafka.log:type=Log,name=Size&lt;/code&gt; → storage per topic-partition&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec&lt;/code&gt; → inbound traffic per topic&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec&lt;/code&gt; → outbound traffic per topic&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Missing Piece: Ownership
&lt;/h2&gt;

&lt;p&gt;Metrics alone aren't enough for chargeback. You also need to know &lt;strong&gt;who owns each topic&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In most Kafka deployments, topic ownership is tribal knowledge. It lives in someone's head, in a Confluence page that's three years out of date, or nowhere at all.&lt;/p&gt;

&lt;p&gt;For chargeback to work, you need a system that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Tracks which team or person owns each topic&lt;/li&gt;
&lt;li&gt;Links cost metrics to that ownership&lt;/li&gt;
&lt;li&gt;Produces a report that finance or engineering management can actually use&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How PartitionPilot Implements This
&lt;/h2&gt;

&lt;p&gt;PartitionPilot connects to your Prometheus endpoint and takes periodic cost snapshots. Each snapshot captures storage and traffic per topic, stamped with a timestamp.&lt;/p&gt;

&lt;p&gt;On top of that, it lets you assign an owner to each topic and consumer group. Ownership is stored in a PostgreSQL database alongside the cost data.&lt;/p&gt;

&lt;p&gt;The result: a chargeback report in CSV format that looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;Owner       | Topic                  | Storage (GB) | Traffic In (GB) | Traffic Out (GB) | Estimated Cost
------------+------------------------+--------------+-----------------+------------------+---------------
Team A      | orders.v2              | 12.4         | 45.2            | 180.8            | CHF 23.40
Team B      | user-events            | 8.1          | 120.3           | 360.9            | CHF 41.20
Team C      | analytics.raw          | 95.2         | 890.1           | 2670.3           | CHF 312.80
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This report can be exported and shared with engineering managers or finance teams on a monthly basis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Is Harder Than It Sounds
&lt;/h2&gt;

&lt;p&gt;A few things make Kafka chargeback tricky in practice:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Topics are shared.&lt;/strong&gt; A single topic can be written to by one team and consumed by three others. Who pays for the outbound traffic — the producer or the consumers? There's no universal answer. PartitionPilot lets you assign separate ownership for producer and consumer sides.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Retention makes storage non-obvious.&lt;/strong&gt; The cost of a topic depends not just on throughput, but on retention settings. A low-traffic topic with 30-day retention can cost more than a high-traffic topic with 1-hour retention.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Metrics need aggregation.&lt;/strong&gt; Raw Prometheus metrics are per-broker, per-partition. You need to aggregate them per topic across all brokers to get meaningful numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;p&gt;PartitionPilot is self-hosted via Docker Compose. You can start a free 30-day trial at &lt;a href="https://partitionpilot.com" rel="noopener noreferrer"&gt;partitionpilot.com&lt;/a&gt; — no credit card required.&lt;/p&gt;

&lt;p&gt;If your team is running Kafka as shared infrastructure and you want to start doing proper cost allocation, give it a try.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Pascal Clément — founder of PartitionPilot&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kafka</category>
      <category>finops</category>
      <category>devops</category>
      <category>cloudcost</category>
    </item>
  </channel>
</rss>
