<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Nimai Charan</title>
    <description>The latest articles on Forem by Nimai Charan (@nimxch).</description>
    <link>https://forem.com/nimxch</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1120860%2Fc6a4cbc3-1072-4cf6-94cc-6f83ecaf5d01.png</url>
      <title>Forem: Nimai Charan</title>
      <link>https://forem.com/nimxch</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/nimxch"/>
    <language>en</language>
    <item>
      <title>HTTP ETag : I got sold by this 90s tech.</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Thu, 22 Jan 2026 17:19:50 +0000</pubDate>
      <link>https://forem.com/nimxch/http-etag-i-got-sold-by-this-90s-tech-59c9</link>
      <guid>https://forem.com/nimxch/http-etag-i-got-sold-by-this-90s-tech-59c9</guid>
      <description>&lt;p&gt;Back in 90s, when modems screamed and images loaded byte by byte, HTTP ETag was born.&lt;/p&gt;

&lt;p&gt;Fast forward to 2026 and &lt;br&gt;
&lt;strong&gt;ETags are still quietly making web faster.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No hype. No fancy frameworks. Just one tiny header doing real work.&lt;/p&gt;

&lt;p&gt;In this AI era, where bandwidth feels unlimited and sloppy code ships daily, ETags are a beautiful piece of engineering that somehow still survives.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Fun Fact: I was unaware of it until I read it from one of the &lt;a href="https://zerodha.tech/blog/scaling-with-common-sense/" rel="noopener noreferrer"&gt;Zerodhas's tech blog&lt;/a&gt; fro 2020. &lt;strong&gt;I tried, I was sold.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What is an HTTP ETag?
&lt;/h2&gt;

&lt;p&gt;ETag = &lt;em&gt;Entity Tag&lt;/em&gt; - a fingerprint for a response.&lt;/p&gt;

&lt;p&gt;Flow in one sentence:&lt;br&gt;
&lt;code&gt;Server sends data + ETag --&amp;gt; client sends it back later --&amp;gt; server replied "304 Not Modified"&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;No response body, no bandwidth just pure  engineering. Simple.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I was struggling with?
&lt;/h2&gt;

&lt;p&gt;In one of my apps, the frontend polls the backend &lt;strong&gt;every 10 seconds&lt;/strong&gt; to check device status. &lt;/p&gt;

&lt;p&gt;Most of the time? Nothing changes.&lt;/p&gt;

&lt;p&gt;Without caching:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Same JSON sent again and again&lt;/li&gt;
&lt;li&gt;Same CPU Work, Same MySQL Query&lt;/li&gt;
&lt;li&gt;Same bandwidth wasted&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Naturally someone asked, "Why not websocket or SSE?"
&lt;/h3&gt;

&lt;p&gt;Because... it overkills.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Data changes rarely&lt;/li&gt;
&lt;li&gt;90% of response are identical&lt;/li&gt;
&lt;li&gt;websocket = connection management, reconnection logic, scaling pain&lt;/li&gt;
&lt;li&gt;SSE = long-lived connection, load balancer cries&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;In other word, I did not needed realtime, I needed &lt;strong&gt;Smarter Polling&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now the frontend sends:&lt;br&gt;
&lt;code&gt;If-None-Match: "abc123"&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Backend answers:&lt;br&gt;
&lt;code&gt;304 Not Modified OR  200 OK + new JSON + New ETag&lt;/code&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;~90% of requests return 304
&lt;/li&gt;
&lt;li&gt;Almost zero bandwidth
&lt;/li&gt;
&lt;li&gt;No open sockets
&lt;/li&gt;
&lt;li&gt;Works everywhere
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same polling. Now polite.&lt;/p&gt;

&lt;p&gt;You can find the implementation &lt;a href="https://github.com/nimxch/http-etag" rel="noopener noreferrer"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thought
&lt;/h2&gt;

&lt;p&gt;ETag isn’t legacy tech. Before adding realtime infra, try the header that’s been solving this problem since 1997.&lt;/p&gt;

&lt;p&gt;Sometimes the best optimization… is already in HTTP.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;NOTE: While inspecting request, please make sure &lt;code&gt;Disble Cache&lt;/code&gt; in network panel is un-checked.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Further Reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag" rel="noopener noreferrer"&gt;https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://datatracker.ietf.org/doc/id/draft-reschke-http-etag-on-write-08.html" rel="noopener noreferrer"&gt;https://datatracker.ietf.org/doc/id/draft-reschke-http-etag-on-write-08.html&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>webdev</category>
      <category>http</category>
      <category>performance</category>
    </item>
    <item>
      <title>HTTP ETag : I got sold by this 90s tech.</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Thu, 22 Jan 2026 17:19:50 +0000</pubDate>
      <link>https://forem.com/nimxch/http-etag-i-got-sold-by-this-90s-tech-2a28</link>
      <guid>https://forem.com/nimxch/http-etag-i-got-sold-by-this-90s-tech-2a28</guid>
      <description>&lt;p&gt;Back in 90s, when modems screamed and images loaded byte by byte, HTTP ETag was born.&lt;/p&gt;

&lt;p&gt;Fast forward to 2026 and &lt;br&gt;
&lt;strong&gt;ETags are still quietly making web faster.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No hype. No fancy frameworks. Just one tiny header doing real work.&lt;/p&gt;

&lt;p&gt;In this AI era, where bandwidth feels unlimited and sloppy code ships daily, ETags are a beautiful piece of engineering that somehow still survives.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Fun Fact: I was unaware of it until I read it from one of the &lt;a href="https://zerodha.tech/blog/scaling-with-common-sense/" rel="noopener noreferrer"&gt;Zerodhas's tech blog&lt;/a&gt; fro 2020. &lt;strong&gt;I tried, I was sold.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What is an HTTP ETag?
&lt;/h2&gt;

&lt;p&gt;ETag = &lt;em&gt;Entity Tag&lt;/em&gt; - a fingerprint for a response.&lt;/p&gt;

&lt;p&gt;Flow in one sentence:&lt;br&gt;
&lt;code&gt;Server sends data + ETag --&amp;gt; client sends it back later --&amp;gt; server replied "304 Not Modified"&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;No response body, no bandwidth just pure  engineering. Simple.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I was struggling with?
&lt;/h2&gt;

&lt;p&gt;In one of my apps, the frontend polls the backend &lt;strong&gt;every 10 seconds&lt;/strong&gt; to check device status. &lt;/p&gt;

&lt;p&gt;Most of the time? Nothing changes.&lt;/p&gt;

&lt;p&gt;Without caching:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Same JSON sent again and again&lt;/li&gt;
&lt;li&gt;Same CPU Work, Same MySQL Query&lt;/li&gt;
&lt;li&gt;Same bandwidth wasted&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Naturally someone asked, "Why not websocket or SSE?"
&lt;/h3&gt;

&lt;p&gt;Because... it overkills.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Data changes rarely&lt;/li&gt;
&lt;li&gt;90% of response are identical&lt;/li&gt;
&lt;li&gt;websocket = connection management, reconnection logic, scaling pain&lt;/li&gt;
&lt;li&gt;SSE = long-lived connection, load balancer cries&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;In other word, I did not needed realtime, I needed &lt;strong&gt;Smarter Polling&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now the frontend sends:&lt;br&gt;
&lt;code&gt;If-None-Match: "abc123"&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Backend answers:&lt;br&gt;
&lt;code&gt;304 Not Modified OR  200 OK + new JSON + New ETag&lt;/code&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;~90% of requests return 304
&lt;/li&gt;
&lt;li&gt;Almost zero bandwidth
&lt;/li&gt;
&lt;li&gt;No open sockets
&lt;/li&gt;
&lt;li&gt;Works everywhere
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same polling. Now polite.&lt;/p&gt;

&lt;p&gt;You can find the implementation &lt;a href="https://github.com/nimxch/http-etag" rel="noopener noreferrer"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thought
&lt;/h2&gt;

&lt;p&gt;ETag isn’t legacy tech. Before adding realtime infra, try the header that’s been solving this problem since 1997.&lt;/p&gt;

&lt;p&gt;Sometimes the best optimization… is already in HTTP.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;NOTE: While inspecting request, please make sure &lt;code&gt;Disble Cache&lt;/code&gt; in network panel is un-checked.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Further Reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag" rel="noopener noreferrer"&gt;https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://datatracker.ietf.org/doc/id/draft-reschke-http-etag-on-write-08.html" rel="noopener noreferrer"&gt;https://datatracker.ietf.org/doc/id/draft-reschke-http-etag-on-write-08.html&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>webdev</category>
      <category>http</category>
      <category>performance</category>
    </item>
    <item>
      <title>Mining Git Internals to Build a Year-in-Review Dashboard</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Mon, 05 Jan 2026 01:37:52 +0000</pubDate>
      <link>https://forem.com/nimxch/mining-git-internals-to-build-a-year-in-review-dashboard-14ej</link>
      <guid>https://forem.com/nimxch/mining-git-internals-to-build-a-year-in-review-dashboard-14ej</guid>
      <description>&lt;p&gt;Every year-end, products like Spotify Wrapped or YouTube Music Wrapped present personalized retrospectives using user data. That idea led to a simple question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Can we generate a similar “year-in-review” for engineers—purely from Git, without relying on GitHub or Bitbucket APIs?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The result was &lt;strong&gt;IQnext Wrapped&lt;/strong&gt;: an internal tool that analyzes raw Git repositories to generate a visual, gamified summary of engineering activity across multiple years.&lt;/p&gt;

&lt;p&gt;This post documents the &lt;strong&gt;architecture, core logic, and Git internals&lt;/strong&gt; behind the project.&lt;/p&gt;

&lt;h2&gt;
  
  
  High-Level Architecture
&lt;/h2&gt;

&lt;p&gt;The system is intentionally simple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Git Repositories
      ↓
Go-based Git Miner
      ↓
Aggregated JSON Files
      ↓
Next.js Visualization Layer

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is &lt;strong&gt;no database&lt;/strong&gt;, &lt;strong&gt;no backend service&lt;/strong&gt;, and &lt;strong&gt;no scheduled jobs&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The entire pipeline runs locally and produces static artifacts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Component Breakdown
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Golang Data Miner
&lt;/h3&gt;

&lt;p&gt;The core of the system is a Go program that analyzes repositories directly via the &lt;code&gt;.git&lt;/code&gt; directory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key characteristics:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Uses &lt;code&gt;go-git/v5&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Clones and analyzes 12 internal repositories&lt;/li&gt;
&lt;li&gt;Traverses &lt;strong&gt;all branches&lt;/strong&gt;, not just &lt;code&gt;main&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Deduplicates commits across branches&lt;/li&gt;
&lt;li&gt;Computes aggregate statistics:

&lt;ul&gt;
&lt;li&gt;Total commits&lt;/li&gt;
&lt;li&gt;Lines added / deleted&lt;/li&gt;
&lt;li&gt;Activity by author&lt;/li&gt;
&lt;li&gt;Activity by weekday and month&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;This miner is effectively the &lt;em&gt;warehouse&lt;/em&gt; of the system.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. JSON as the Data Ledger
&lt;/h3&gt;

&lt;p&gt;After analysis, all computed metrics are written to JSON files.&lt;/p&gt;

&lt;p&gt;This design decision was deliberate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Flat files simplify development&lt;/li&gt;
&lt;li&gt;Zero operational overhead&lt;/li&gt;
&lt;li&gt;Easy consumption by frontend&lt;/li&gt;
&lt;li&gt;Immutable, auditable snapshots&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The JSON files act as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The database&lt;/li&gt;
&lt;li&gt;The API&lt;/li&gt;
&lt;li&gt;The single source of truth&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Frontend: Next.js as the Storyteller
&lt;/h3&gt;

&lt;p&gt;The UI is built using &lt;strong&gt;Next.js 14 + TypeScript&lt;/strong&gt; and consumes the generated JSON directly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key features:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Year-over-year comparison&lt;/li&gt;
&lt;li&gt;Team rankings and consistency scores&lt;/li&gt;
&lt;li&gt;Achievement badges (e.g., &lt;em&gt;Early Bird&lt;/em&gt;, &lt;em&gt;Weekend Warrior&lt;/em&gt;)&lt;/li&gt;
&lt;li&gt;Framer Motion for transitions and animations&lt;/li&gt;
&lt;li&gt;Session-based authentication for internal access&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The frontend does no computation—it purely visualizes precomputed data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Core Insight: The &lt;code&gt;.git&lt;/code&gt; Directory Is a Database
&lt;/h2&gt;

&lt;p&gt;Most developers treat &lt;code&gt;.git&lt;/code&gt; as an opaque implementation detail. For this project, it is the primary data source.&lt;/p&gt;

&lt;p&gt;Understanding Git internals was critical.&lt;/p&gt;

&lt;h3&gt;
  
  
  Entry Points: &lt;code&gt;.git/HEAD&lt;/code&gt; and &lt;code&gt;.git/refs&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Each branch reference ultimately points to a commit hash.&lt;/p&gt;

&lt;p&gt;By iterating through references:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All branches become visible&lt;/li&gt;
&lt;li&gt;All reachable commits can be collected&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This ensures contributions are captured regardless of where they occurred.&lt;/p&gt;

&lt;h3&gt;
  
  
  Storage Layer: &lt;code&gt;.git/objects&lt;/code&gt; and the Commit DAG
&lt;/h3&gt;

&lt;p&gt;Git stores data as a &lt;strong&gt;content-addressable object graph&lt;/strong&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Loose objects (&lt;code&gt;.git/objects/??/&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Packfiles (&lt;code&gt;.git/objects/pack/&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Objects are compressed using zlib&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Using &lt;code&gt;repo.CommitObjects()&lt;/code&gt;, &lt;code&gt;go-git&lt;/code&gt; transparently:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Decompresses objects&lt;/li&gt;
&lt;li&gt;Reconstructs commits, trees, and blobs&lt;/li&gt;
&lt;li&gt;Exposes them as Go structs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This enables full traversal of the &lt;strong&gt;Directed Acyclic Graph (DAG)&lt;/strong&gt; of commits.&lt;/p&gt;

&lt;h3&gt;
  
  
  Understanding Change: Parents, Trees, and Diffs
&lt;/h3&gt;

&lt;p&gt;Each commit points to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One or more parent commits&lt;/li&gt;
&lt;li&gt;A tree representing a snapshot of the repository&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To compute line statistics:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Compare a commit to its parent&lt;/li&gt;
&lt;li&gt;Diff the parent tree against the current tree&lt;/li&gt;
&lt;li&gt;Drill down to blob-level differences&lt;/li&gt;
&lt;li&gt;Count additions and deletions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is how metrics like “lines added” and “lines removed” are derived—without any external tooling.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Commit Analyzer
&lt;/h2&gt;

&lt;p&gt;All collected data funnels into a central structure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;RepositoryStats&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Name&lt;/span&gt;              &lt;span class="kt"&gt;string&lt;/span&gt;           &lt;span class="s"&gt;`json:"name"`&lt;/span&gt;
    &lt;span class="n"&gt;TotalCommits&lt;/span&gt;      &lt;span class="kt"&gt;int&lt;/span&gt;              &lt;span class="s"&gt;`json:"total_commits"`&lt;/span&gt;
    &lt;span class="n"&gt;TotalLinesAdded&lt;/span&gt;   &lt;span class="kt"&gt;int&lt;/span&gt;              &lt;span class="s"&gt;`json:"total_lines_added"`&lt;/span&gt;
    &lt;span class="n"&gt;TotalLinesDeleted&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;              &lt;span class="s"&gt;`json:"total_lines_deleted"`&lt;/span&gt;
    &lt;span class="n"&gt;CommitsByAuthor&lt;/span&gt;   &lt;span class="k"&gt;map&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;   &lt;span class="s"&gt;`json:"commits_by_author"`&lt;/span&gt;
    &lt;span class="n"&gt;CommitsByMonth&lt;/span&gt;    &lt;span class="k"&gt;map&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;   &lt;span class="s"&gt;`json:"commits_by_month"`&lt;/span&gt;
    &lt;span class="n"&gt;CommitsByWeekday&lt;/span&gt;  &lt;span class="k"&gt;map&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;   &lt;span class="s"&gt;`json:"commits_by_weekday"`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To avoid double-counting commits across branches, the analyzer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Iterates through all reachable commits&lt;/li&gt;
&lt;li&gt;Tracks seen commit hashes&lt;/li&gt;
&lt;li&gt;Aggregates metrics only once per unique commit
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;allCommitHashes&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="n"&gt;plumbing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Hash&lt;/span&gt;

&lt;span class="n"&gt;branchIter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ForEach&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;object&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Commit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;seenCommits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Hash&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;seenCommits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Hash&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
        &lt;span class="n"&gt;allCommitHashes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;allCommitHashes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Hash&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This guarantees a complete and accurate view of contribution history.&lt;/p&gt;

&lt;h2&gt;
  
  
  Design Constraint: One Core File
&lt;/h2&gt;

&lt;p&gt;The entire backend logic lives in &lt;strong&gt;one file: &lt;code&gt;main.go&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Everything else—frontend, animations, UI polish—is secondary.&lt;/p&gt;

&lt;p&gt;This constraint forced:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Clear data flow&lt;/li&gt;
&lt;li&gt;Minimal abstractions&lt;/li&gt;
&lt;li&gt;Easier reasoning and debugging&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lessons Learned
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Git already contains everything&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;APIs are convenience layers, not requirements&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flat files are underrated&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;JSON works extremely well for read-heavy analytics&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Branch-aware analytics matter&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;main&lt;/code&gt;only analysis hides real work&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Understanding internals pays off&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Treating Git as a database unlocks new possibilities&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;IQnext Wrapped started as a side project inspired by year-end review products, but became a deep dive into Git internals, data modeling, and disciplined architecture.&lt;/p&gt;

&lt;p&gt;The project reinforced a simple idea:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Many problems do not need more infrastructure—only better understanding of the systems we already use.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The Go-based Git analyzer is robust, the frontend is expressive, and most importantly, the system remains simple, inspectable, and dependency-free.&lt;/p&gt;

</description>
      <category>go</category>
      <category>git</category>
      <category>programming</category>
    </item>
    <item>
      <title>Designing a Lightweight Data Pipeline for Analysis Without Stressing Production Databases</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Mon, 22 Dec 2025 15:05:47 +0000</pubDate>
      <link>https://forem.com/nimxch/designing-a-lightweight-data-pipeline-for-analysis-without-stressing-production-databases-53bh</link>
      <guid>https://forem.com/nimxch/designing-a-lightweight-data-pipeline-for-analysis-without-stressing-production-databases-53bh</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;Analytics should never compete with production workloads.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In many systems, analytics begin as a convenience query against the production database. Over time, these queries grow heavier, scan larger time ranges, and slowly degrade system reliability.&lt;/p&gt;

&lt;p&gt;This post describes a &lt;strong&gt;lightweight, production-safe data pipeline for analytical workloads&lt;/strong&gt;, built with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Delta Lake on S3&lt;/strong&gt; for durable analytical storage&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;DuckDB&lt;/strong&gt; as an embedded analytics engine&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Tenant-isolated batch processing&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Strict resource limits&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Local caching to reduce cloud IO&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is &lt;strong&gt;not a streaming system&lt;/strong&gt; and &lt;strong&gt;not an anomaly engine&lt;/strong&gt; yet.&lt;br&gt;&lt;br&gt;
It is a &lt;strong&gt;clean analytical data pipeline&lt;/strong&gt; designed to protect production databases.&lt;/p&gt;
&lt;h3&gt;
  
  
  The Problem With Analytics on Production Databases
&lt;/h3&gt;

&lt;p&gt;Production databases are optimized for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Fast point reads&lt;/li&gt;
&lt;li&gt;  High write concurrency&lt;/li&gt;
&lt;li&gt;  Transactional guarantees&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Also they are &lt;strong&gt;not optimized&lt;/strong&gt; for operation like &lt;strong&gt;large historical scans&lt;/strong&gt;,  &lt;strong&gt;aggregations over months or years&lt;/strong&gt;, &lt;strong&gt;repeated analytical jobs&lt;/strong&gt; or &lt;strong&gt;heavy read amplification&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Running analytics on production systems leads to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Query latency spikes&lt;/li&gt;
&lt;li&gt;  Lock contention&lt;/li&gt;
&lt;li&gt;  CPU starvation&lt;/li&gt;
&lt;li&gt;  Unpredictable failures during peak hours&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The primary architectural goal was simple:&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Never run analytical queries on the production database.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;
  
  
  Architectural Principles
&lt;/h3&gt;

&lt;p&gt;This pipeline is built on four non-negotiable principles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Production isolation&lt;/strong&gt; – analytics never touch OLTP systems
&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Batch-first design&lt;/strong&gt; – no streaming complexity&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Bounded resources&lt;/strong&gt; – strict memory and thread limits&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Operational simplicity&lt;/strong&gt; – no clusters to manage&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;
  
  
  High-Level Architecture
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4267qypcl8yvhm9fau4e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4267qypcl8yvhm9fau4e.png" alt=" " width="800" height="335"&gt;&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;DuckDB runs inside the application process&lt;/strong&gt;, not as a service.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why Delta Lake on S3?
&lt;/h3&gt;

&lt;p&gt;Delta Lake provides ACID transactions on object storage, schema enforcement, safe concurrent writes anf features like time-travel for reprocessing. &lt;/p&gt;

&lt;p&gt;In the other hand &lt;strong&gt;Amazon S3&lt;/strong&gt; have low cost storage and high durability.&lt;/p&gt;

&lt;p&gt;This combination creates a &lt;strong&gt;stable analytical backbone&lt;/strong&gt; that is fully decoupled from production systems.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why DuckDB?
&lt;/h3&gt;

&lt;p&gt;DuckDB is an &lt;strong&gt;embedded OLAP engine&lt;/strong&gt;, not a distributed system.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why that matters:
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;  No JVM&lt;/li&gt;
&lt;li&gt;  No cluster&lt;/li&gt;
&lt;li&gt;  No coordinator&lt;/li&gt;
&lt;li&gt;  No operational overhead&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Apart from this DuckDB excels at &lt;strong&gt;columnar scans&lt;/strong&gt;, &lt;strong&gt;vectorized execution&lt;/strong&gt;.  And also it is know for high Parquet / Delta reads and batch analytical workloads.&lt;/p&gt;

&lt;p&gt;This makes it ideal for scheduled jobs and tenant-isolated analytics. Also we can make resource-constrained environments with it.&lt;/p&gt;
&lt;h3&gt;
  
  
  Explicit Resource Control
&lt;/h3&gt;

&lt;p&gt;DuckDB is configured with &lt;strong&gt;hard limits&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;memory_limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'1GB'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="k"&gt;As&lt;/span&gt; &lt;span class="n"&gt;per&lt;/span&gt; &lt;span class="n"&gt;environment&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;threads&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="n"&gt;Based&lt;/span&gt; &lt;span class="n"&gt;upon&lt;/span&gt; &lt;span class="n"&gt;available&lt;/span&gt; &lt;span class="n"&gt;CPU&lt;/span&gt; &lt;span class="n"&gt;cores&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_object_cache&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This guarantees &lt;strong&gt;no memory explosions&lt;/strong&gt; and a predictable performance.&lt;/p&gt;

&lt;p&gt;Each analytical run behaves like a &lt;strong&gt;bounded analytical task&lt;/strong&gt;, not a long-running service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tenant-Isolated Data Layout
&lt;/h3&gt;

&lt;p&gt;Data is stored per tenant:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;s3://data-lake/
  tenant_id=tenant_123/
    energy_reading/
      year=2025/
        month=03/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Benefits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Partition pruning&lt;/li&gt;
&lt;li&gt;  Reduced S3 reads&lt;/li&gt;
&lt;li&gt;  Strong tenant isolation&lt;/li&gt;
&lt;li&gt;  Linear horizontal scaling by tenant&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each job processes &lt;strong&gt;one tenant at a time&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scheduler-Based Processing (Not Streaming)
&lt;/h3&gt;

&lt;p&gt;This system intentionally avoids streaming frameworks.&lt;/p&gt;

&lt;p&gt;Why:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Analytics are not latency-critical&lt;/li&gt;
&lt;li&gt;  Batch jobs are deterministic&lt;/li&gt;
&lt;li&gt;  Failures are easier to retry&lt;/li&gt;
&lt;li&gt;  Infrastructure cost is dramatically lower&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Jobs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Run on a scheduler&lt;/li&gt;
&lt;li&gt;  Read bounded time ranges&lt;/li&gt;
&lt;li&gt;  Exit cleanly after execution&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Local Disk Cache to Reduce S3 Reads
&lt;/h3&gt;

&lt;p&gt;Repeated S3 scans might be  expensive and slow.&lt;/p&gt;

&lt;p&gt;A local disk cache:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Stores tenant-scoped results&lt;/li&gt;
&lt;li&gt;  Uses TTL-based eviction&lt;/li&gt;
&lt;li&gt;  Eliminates redundant reads&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This significantly improves &lt;strong&gt;latency&lt;/strong&gt;,  &lt;strong&gt;Cost efficiency&lt;/strong&gt; and overall system stability.&lt;/p&gt;

&lt;h3&gt;
  
  
  Performance Benchmarks (DuckDB + Delta Lake on S3)
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Environment&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Instance: 4 vCPU / 8 GB RAM
&lt;/li&gt;
&lt;li&gt;DuckDB memory limit: 1 GB
&lt;/li&gt;
&lt;li&gt;DuckDB threads: 2
&lt;/li&gt;
&lt;li&gt;Storage: Amazon S3 (Delta Lake)
&lt;/li&gt;
&lt;li&gt;Cache: Local disk (result-level cache)
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Cold Read (S3 + Delta Scan)&lt;/th&gt;
&lt;th&gt;Warm Read (Local Cache)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rows scanned&lt;/td&gt;
&lt;td&gt;~2.1 million&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rows returned&lt;/td&gt;
&lt;td&gt;~120,000&lt;/td&gt;
&lt;td&gt;~120,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data read from S3&lt;/td&gt;
&lt;td&gt;~180 MB&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Query execution time&lt;/td&gt;
&lt;td&gt;4.8 – 6.2 seconds&lt;/td&gt;
&lt;td&gt;40 – 90 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peak memory usage&lt;/td&gt;
&lt;td&gt;~620 MB&lt;/td&gt;
&lt;td&gt;~120 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CPU utilization&lt;/td&gt;
&lt;td&gt;~1.5 – 1.8 cores&lt;/td&gt;
&lt;td&gt;&amp;lt; 0.3 core&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network I/O&lt;/td&gt;
&lt;td&gt;High (S3 reads)&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Production DB load&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Multi-Tenant Sequential Jobs
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Number of Tenants&lt;/th&gt;
&lt;th&gt;Total Processing Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;10 tenants&lt;/td&gt;
&lt;td&gt;~45 seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;25 tenants&lt;/td&gt;
&lt;td&gt;~2 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50 tenants&lt;/td&gt;
&lt;td&gt;~4 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Observation:&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
No cross-tenant interference was observed. Each tenant job executed in isolation with predictable and linear scaling characteristics.&lt;/p&gt;

&lt;h3&gt;
  
  
  Comparison With ClickHouse and Spark
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Engine&lt;/th&gt;
&lt;th&gt;Best For&lt;/th&gt;
&lt;th&gt;Strengths&lt;/th&gt;
&lt;th&gt;Tradeoffs / Downsides&lt;/th&gt;
&lt;th&gt;Operational Complexity&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;DuckDB (This Approach)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Embedded, batch analytics&lt;/td&gt;
&lt;td&gt;- Runs in-process&lt;br&gt;- No cluster management&lt;br&gt;- Strong S3 + Parquet support&lt;br&gt;- Predictable resource usage&lt;/td&gt;
&lt;td&gt;- Single-node execution&lt;br&gt;- Not designed for high concurrent queries&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Low&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;ClickHouse&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Real-time analytical services&lt;/td&gt;
&lt;td&gt;- Extremely fast OLAP queries&lt;br&gt;- High query concurrency&lt;br&gt;- Mature production deployments&lt;/td&gt;
&lt;td&gt;- Requires dedicated servers&lt;br&gt;- Stateful storage management&lt;br&gt;- Higher ops overhead&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Medium–High&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Apache Spark&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Large-scale data processing&lt;/td&gt;
&lt;td&gt;- Horizontally scalable&lt;br&gt;- Excellent for ETL and ML workloads&lt;br&gt;- Rich ecosystem&lt;/td&gt;
&lt;td&gt;- Heavy memory footprint&lt;br&gt;- Slow startup time&lt;br&gt;- Significant operational complexity&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;High&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Takeaway:&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
For bounded, tenant-isolated analytical jobs running on a schedule, DuckDB provides the best balance of performance, cost, and operational simplicity.&lt;/p&gt;

&lt;h3&gt;
  
  
  What This Pipeline Is (and What It Is Not)
&lt;/h3&gt;

&lt;p&gt;This pipeline is designed to solve a very specific problem: running analytical workloads without putting any pressure on production systems.&lt;/p&gt;

&lt;p&gt;It acts as a &lt;strong&gt;production-safe analytical layer&lt;/strong&gt;, allowing historical data to be scanned, aggregated, and analyzed in isolation. It also serves as a clean foundation for downstream logic such as anomaly detection or reporting, without forcing those concerns into the storage or query layer. Most importantly, it achieves this without requiring a dedicated analytics cluster, making it a cost-efficient alternative for teams operating under infrastructure constraints.&lt;/p&gt;

&lt;p&gt;At the same time, this pipeline is intentionally narrow in scope.&lt;/p&gt;

&lt;p&gt;It is not a streaming system, and it does not aim to deliver real-time insights. It is also not an OLTP replacement, nor does it embed anomaly detection logic directly. The analytical engine focuses purely on reliable, bounded data access — any anomaly detection or intelligence logic is built on top of it, not baked into it.&lt;/p&gt;

&lt;p&gt;This separation keeps the system easier to reason about, test, and evolve over time.&lt;/p&gt;

&lt;h3&gt;
  
  
  Final Thoughts
&lt;/h3&gt;

&lt;p&gt;Not every analytical problem requires a cluster, a streaming framework, or a heavyweight OLAP database.&lt;/p&gt;

&lt;p&gt;In many cases, especially for scheduled or batch-driven analysis, those tools introduce more operational complexity than value. A simpler system — one that is bounded, predictable, and easy to operate — often performs better in practice.&lt;/p&gt;

&lt;p&gt;By combining DuckDB with Delta Lake on S3, this pipeline behaves like an analytical sidecar: it stays out of the way of production workloads while still delivering fast, reliable analysis when needed.&lt;/p&gt;

&lt;p&gt;The key takeaway is not about the tools themselves, but about architectural restraint. Choosing the simplest system that meets your requirements often leads to more stable, maintainable, and cost-effective outcomes.&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>database</category>
      <category>python</category>
      <category>aws</category>
    </item>
    <item>
      <title>Building a Meta-Logger: Tracking My Work Across GitHub, Codeberg, and Bitbucket Using Go</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Thu, 18 Dec 2025 15:22:23 +0000</pubDate>
      <link>https://forem.com/nimxch/building-a-meta-logger-tracking-my-work-across-github-codeberg-and-bitbucket-using-go-4kp6</link>
      <guid>https://forem.com/nimxch/building-a-meta-logger-tracking-my-work-across-github-codeberg-and-bitbucket-using-go-4kp6</guid>
      <description>&lt;p&gt;Modern development isn’t just fragmented across repositories — it’s fragmented across &lt;strong&gt;platforms&lt;/strong&gt;.&lt;br&gt;
On any given week, I might:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Fix a production bug in a &lt;strong&gt;Bitbucket&lt;/strong&gt; repo at work&lt;/li&gt;
&lt;li&gt;  Push an open-source contribution to &lt;strong&gt;Codeberg&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;  Ship a side-project feature on &lt;strong&gt;GitHub&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each platform has its own dashboards, contribution graphs, and activity feeds.&lt;br&gt;&lt;br&gt;
None of them talk to each other.&lt;/p&gt;

&lt;p&gt;After a while, I noticed a real problem:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I had no reliable way to answer &lt;em&gt;“What have I actually contributed, where, and how much?”&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Not because the data didn’t exist — but because it was &lt;strong&gt;siloed&lt;/strong&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  The Problem: Contribution Blindness
&lt;/h2&gt;

&lt;p&gt;This wasn’t about chasing green squares or streaks.&lt;br&gt;
It was about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Losing visibility into my own work&lt;/li&gt;
&lt;li&gt;  Forgetting entire weeks of meaningful contributions&lt;/li&gt;
&lt;li&gt;  Having no portable, platform-agnostic record of progress&lt;/li&gt;
&lt;li&gt;  Manually reconstructing timelines whenever I needed a summary&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When someone asked:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“What did you work on last month?”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The answer involved:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Opening GitHub&lt;/li&gt;
&lt;li&gt;  Then Bitbucket&lt;/li&gt;
&lt;li&gt;  Then Codeberg&lt;/li&gt;
&lt;li&gt;  Searching commit histories&lt;/li&gt;
&lt;li&gt;  Mentally stitching together context&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That’s a tooling failure.&lt;/p&gt;
&lt;h2&gt;
  
  
  The Insight: Git Is the Only Common Denominator
&lt;/h2&gt;

&lt;p&gt;Platforms differ.&lt;br&gt;&lt;br&gt;
Git does not.&lt;br&gt;
Whether a repo lives on GitHub, Codeberg, Bitbucket, or a self-hosted server, every commit has:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A hash&lt;/li&gt;
&lt;li&gt;  An author&lt;/li&gt;
&lt;li&gt;  A timestamp&lt;/li&gt;
&lt;li&gt;  A message&lt;/li&gt;
&lt;li&gt;  A repository context&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So instead of trying to unify &lt;strong&gt;platforms&lt;/strong&gt;, I decided to unify &lt;strong&gt;my commits&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That led to a simple idea:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What if every commit I make, anywhere, automatically logs itself into one central place?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;
  
  
  What I Built: A Meta-Logger
&lt;/h2&gt;

&lt;p&gt;I built a small Go-based CLI tool I call a &lt;strong&gt;Meta Logger&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It creates a &lt;strong&gt;meta-repository&lt;/strong&gt; — not for code, but for &lt;strong&gt;context&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This repository becomes&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A centralized journal of my work&lt;/li&gt;
&lt;li&gt;  A searchable, chronological history&lt;/li&gt;
&lt;li&gt;  A platform-independent “proof of work”&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The tool works silently in the background and requires zero manual effort once set up.&lt;/p&gt;
&lt;h2&gt;
  
  
  Architecture Overview
&lt;/h2&gt;

&lt;p&gt;The system has three moving parts:&lt;/p&gt;
&lt;h3&gt;
  
  
  1. The Log Repository
&lt;/h3&gt;

&lt;p&gt;A dedicated Git repository (hosted anywhere — GitHub is just convenient) that stores:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A Markdown file (&lt;code&gt;commit-log.md&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;  One entry per commit, across all projects&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It does &lt;strong&gt;not&lt;/strong&gt; store source code.&lt;/p&gt;
&lt;h3&gt;
  
  
  2. A Go CLI Tool
&lt;/h3&gt;

&lt;p&gt;A single binary written in Go that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Extracts commit metadata&lt;/li&gt;
&lt;li&gt;  Formats human-readable log entries&lt;/li&gt;
&lt;li&gt;  Commits and syncs them to the log repository&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No runtime dependencies. No daemon. No background service.&lt;/p&gt;
&lt;h3&gt;
  
  
  3. Git Hooks
&lt;/h3&gt;

&lt;p&gt;A &lt;code&gt;post-commit&lt;/code&gt; hook installed into each working repository.&lt;/p&gt;

&lt;p&gt;Every time I run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git commit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logger runs automatically.&lt;/p&gt;

&lt;p&gt;No extra commands. No discipline required.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Go?
&lt;/h2&gt;

&lt;p&gt;I chose Go for two very practical reasons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Speed&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
This runs on every commit. If it’s slow or flaky, it’s dead. Go binaries start instantly.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Zero Dependencies&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
One compiled binary. No Node. No Python. No virtualenvs. No &lt;code&gt;node_modules&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The commit workflow should be invisible.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Works in Practice
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Setup
&lt;/h3&gt;

&lt;p&gt;Running:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go run main.go setup
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Does three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; Initializes a central log repository&lt;/li&gt;
&lt;li&gt; Creates a Markdown log file&lt;/li&gt;
&lt;li&gt; Injects a &lt;code&gt;post-commit&lt;/code&gt; hook into selected repositories&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The hook is minimal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;cd&lt;/span&gt; /path/to/workspace
go run main.go log-commit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From this point on, logging is automatic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Automatic, Invisible Logging
&lt;/h2&gt;

&lt;p&gt;On every commit, the tool captures:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Repository (service) name&lt;/li&gt;
&lt;li&gt;  Branch&lt;/li&gt;
&lt;li&gt;  Commit hash&lt;/li&gt;
&lt;li&gt;  Author&lt;/li&gt;
&lt;li&gt;  Original commit date&lt;/li&gt;
&lt;li&gt;  Full commit message&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And appends a structured Markdown entry like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## [dummy-service] a1b2c3d - 2025-05-10 14:30:00&lt;/span&gt;
&lt;span class="p"&gt;
-&lt;/span&gt; &lt;span class="gs"&gt;**Service:**&lt;/span&gt; dummy-service
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="gs"&gt;**Branch:**&lt;/span&gt; main
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="gs"&gt;**Commit:**&lt;/span&gt; a1b2c3d
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="gs"&gt;**Author:**&lt;/span&gt; Nimai
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="gs"&gt;**Date:**&lt;/span&gt; 2025-05-10 14:29:58
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="gs"&gt;**Message:**&lt;/span&gt; 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This lives in a single file that GitHub (or any Git host) makes instantly searchable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Importing My Entire Past
&lt;/h2&gt;

&lt;p&gt;Logging forward wasn’t enough.&lt;br&gt;&lt;br&gt;
I wanted my &lt;strong&gt;entire history&lt;/strong&gt; reflected.&lt;/p&gt;

&lt;p&gt;That’s what &lt;code&gt;import-history&lt;/code&gt; does.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go run main.go import-history
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  Iterates through all repositories&lt;/li&gt;
&lt;li&gt;  Filters commits by author&lt;/li&gt;
&lt;li&gt;  Reconstructs entries in chronological order&lt;/li&gt;
&lt;li&gt;  Preserves &lt;strong&gt;original commit dates&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Preserving Time Correctly
&lt;/h3&gt;

&lt;p&gt;Git normally assigns &lt;em&gt;now&lt;/em&gt; as the commit time.&lt;br&gt;&lt;br&gt;
That would destroy historical accuracy.&lt;/p&gt;

&lt;p&gt;So while committing to the log repo, I explicitly inject the original dates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Env&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Environ&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="s"&gt;"GIT_AUTHOR_DATE="&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;commitDate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s"&gt;"GIT_COMMITTER_DATE="&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;commitDate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result: the log repository becomes a faithful mirror of my actual work timeline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Silent Syncing by Design
&lt;/h2&gt;

&lt;p&gt;One important design choice: &lt;strong&gt;never block the developer&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If GitHub (or any remote) is unreachable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  The log file is still updated locally&lt;/li&gt;
&lt;li&gt;  The push is attempted silently&lt;/li&gt;
&lt;li&gt;  Failures are ignored&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The next successful push syncs everything naturally.&lt;br&gt;
No broken workflows. No lost commits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Works Across Platforms
&lt;/h2&gt;

&lt;p&gt;This system doesn’t care whether a repository lives on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  GitHub&lt;/li&gt;
&lt;li&gt;  Codeberg&lt;/li&gt;
&lt;li&gt;  Bitbucket&lt;/li&gt;
&lt;li&gt;  Or a local Git server&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As long as it’s Git, it’s logged.&lt;/p&gt;

&lt;p&gt;The log repository itself can live anywhere — GitHub just happens to give a great UI for browsing Markdown.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Gave Me
&lt;/h2&gt;

&lt;p&gt;After a few weeks of usage, I had:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A single, chronological journal of my work&lt;/li&gt;
&lt;li&gt;  A platform-agnostic contribution record&lt;/li&gt;
&lt;li&gt;  A way to answer &lt;em&gt;“What did I work on?”&lt;/em&gt; instantly&lt;/li&gt;
&lt;li&gt;  A clear signal of consistency and growth over time&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No dashboards. No metrics. Just facts.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Is &lt;em&gt;Not&lt;/em&gt;
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Not a productivity tracker&lt;/li&gt;
&lt;li&gt;  Not a replacement for GitHub contributions&lt;/li&gt;
&lt;li&gt;  Not a SaaS product&lt;/li&gt;
&lt;li&gt;  Not something that spies on you&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It’s just Git, documenting Git.&lt;/p&gt;

&lt;h2&gt;
  
  
  Closing Thoughts
&lt;/h2&gt;

&lt;p&gt;This tool changed how I perceive my own output.&lt;/p&gt;

&lt;p&gt;Instead of fragmented activity scattered across platforms, I now have a &lt;strong&gt;unified narrative&lt;/strong&gt; of my work — one that I own completely.&lt;/p&gt;

&lt;p&gt;Sometimes the most useful tools aren’t new abstractions.&lt;/p&gt;

&lt;p&gt;They’re just honest layers on top of systems that already work.&lt;/p&gt;

</description>
      <category>programming</category>
      <category>go</category>
      <category>git</category>
    </item>
    <item>
      <title>EP-1: WTF is Spring Boot?</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Sat, 01 Jun 2024 19:43:50 +0000</pubDate>
      <link>https://forem.com/nimxch/ep-1-wtf-is-spring-boot-a31</link>
      <guid>https://forem.com/nimxch/ep-1-wtf-is-spring-boot-a31</guid>
      <description>&lt;h2&gt;
  
  
  The Man, The Myth, Not Thala, But Spring Boot
&lt;/h2&gt;

&lt;p&gt;What make a java developer powerful? Obviously knowing how to write REST API. Having ability to talk to server by writing mere alphabets in your code editor is not less than having any superpower. In the world of Java, Its not the REDBULL which gives you wing, Its &lt;strong&gt;Spring Boot&lt;/strong&gt; which gives you wings and let you fly high.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to have super power
&lt;/h2&gt;

&lt;p&gt;No individual born with power, people have to acquire it. If you want to have the super power of talking to server you need to go to a place in internet &lt;a href="https://start.spring.io/" rel="noopener noreferrer"&gt;start.spring.io&lt;/a&gt;. The virtual &lt;strong&gt;pornhub ** for java developers.&lt;br&gt;
**What it gives&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Boilerplate starter code for creating a basic spring boot application.&lt;/li&gt;
&lt;li&gt;Standard project structure, Ensures your code can suck but not your project structure&lt;/li&gt;
&lt;li&gt;Nothing else&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F913o5jralq22ou0tflnq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F913o5jralq22ou0tflnq.png" alt="Snip of how start.spring.io looks like" width="800" height="473"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you hit on the GENERATE button one zip folder will magically get downloaded.&lt;br&gt;
Unzip It, Open it with some of the world class IDE like Intellij Idea or VS code and hit on the Play button.&lt;/p&gt;

&lt;p&gt;DONE, You have created your first Spring boot project.&lt;/p&gt;

&lt;p&gt;Lets DIVE Deep on next Episode. &lt;/p&gt;

&lt;p&gt;Happy Booting.&lt;/p&gt;

</description>
      <category>java</category>
      <category>springboot</category>
      <category>beginners</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Danni, Johny and the Java Feature Flag Problem</title>
      <dc:creator>Nimai Charan</dc:creator>
      <pubDate>Mon, 31 Jul 2023 08:34:29 +0000</pubDate>
      <link>https://forem.com/nimxch/danni-johny-and-the-java-toggle-problem-28h0</link>
      <guid>https://forem.com/nimxch/danni-johny-and-the-java-toggle-problem-28h0</guid>
      <description>&lt;h2&gt;
  
  
  What is Feature Toggle
&lt;/h2&gt;

&lt;p&gt;Imagine you are building a new feature for your Java application. You want to test the feature with a small group of users before you roll it out to everyone. You could do this by creating two versions of your application, one with the new feature enabled and one with the new feature disabled. But this would require you to deploy two different versions of your application to production, which could be time-consuming and difficult.&lt;/p&gt;

&lt;p&gt;Instead, you could use a &lt;strong&gt;feature toggle&lt;/strong&gt;. A feature toggle is a variable that you can use to enable or disable a feature in your application at runtime. This means that you can deploy a single version of your application to production, and then use the feature toggle to control whether the new feature is enabled for different users.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Danny and Johny Problem
&lt;/h2&gt;

&lt;p&gt;Imagine a situation &lt;strong&gt;Danny&lt;/strong&gt; is the only daughter of &lt;strong&gt;Big-B&lt;/strong&gt; for which Danny's father &lt;br&gt;
is too protective of her. He never allowed Danny for party or any outing. He works in an hospital as a doctor on shift basis and its not certain when he will go to hospital and come back.  Danny has a boyfriend &lt;strong&gt;Johny&lt;/strong&gt; who is too much partyholic  and he never gots time to chill-out with danny. So, together they found out a solution. They bribed a &lt;strong&gt;Ramu Kaka&lt;/strong&gt; who runs tea stall in front of hospital, who will inform wheather Big-B is in hospital or left for hoe, so that in that time both Johny and Danny can party together.&lt;/p&gt;

&lt;p&gt;Let's Understand chronology&lt;/p&gt;

&lt;blockquote&gt;
&lt;ol&gt;
&lt;li&gt;PAPA_HOME is the feature that &lt;strong&gt;Johny&lt;/strong&gt; and &lt;strong&gt;Danny&lt;/strong&gt; are dependent&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Johny&lt;/strong&gt; will call &lt;strong&gt;Danny&lt;/strong&gt; for party&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Danny&lt;/strong&gt; will be keep looking for &lt;strong&gt;Ramu Kaka&lt;/strong&gt;'s message.&lt;/li&gt;
&lt;li&gt;They will decide if they will go for party or not based on &lt;strong&gt;Ramu Kaka&lt;/strong&gt;'s message.&lt;/li&gt;
&lt;/ol&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flkhw3039btfqssuxnwpw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flkhw3039btfqssuxnwpw.png" alt="Problem image" width="624" height="521"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Solution
&lt;/h2&gt;

&lt;p&gt;For the solution we have several libraries available, I went with &lt;strong&gt;FF4J&lt;/strong&gt;(Feature Flag for java). Below are the Code Snippet with Explanation.&lt;/p&gt;
&lt;h5&gt;
  
  
  1. Head to &lt;a href="https://start.spring.io/" rel="noopener noreferrer"&gt;Spring Intializer&lt;/a&gt; and create a starter spring boot project.
&lt;/h5&gt;

&lt;blockquote&gt;
&lt;p&gt;I don't give a sh*t about which version you are using, But if you are using Spring Boot 3.x you have to use Java 17 or above. Gradle or Maven? I don't care. I went with Gradle.&lt;/p&gt;
&lt;h5&gt;
  
  
  2. Inject the following line to your dependency for gradle.
&lt;/h5&gt;

&lt;p&gt;&lt;code&gt;implementation 'org.ff4j:ff4j-core:2.0.0'&lt;/code&gt;. &lt;br&gt;
If you are using maven, go to &lt;a href="https://mvnrepository.com/artifact/org.ff4j/ff4j-core" rel="noopener noreferrer"&gt;Maven Central Repository&lt;/a&gt; and find your own dependency,&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h5&gt;
  
  
  3. Create A Java configuration something like FF4JConfig or whatever the f*ck you want.
&lt;/h5&gt;

&lt;blockquote&gt;
&lt;p&gt;Where to create? This piece of sh*t is not for you. &lt;/p&gt;
&lt;/blockquote&gt;
&lt;h5&gt;
  
  
  4. Add the following lines.
&lt;/h5&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@Configuration //This piece of shit will tell spring boot that this is a configuration class
public class FF4jConfig {
    //Feature Name Definition
    public static final String PAPA_FEATURE = "PAPA_FEATURE";

    /*
    * FF4J Configuration
    * Create one bean of type FF4J. Why? Because concept of overriding is dying
    * Create of instance of FF4j. Obviously to use it further
    * Register your Feature
    * Enable or disable according to you
    * You are done, Thank me later (Code Below ⬇️)_
    */
    @Bean
    public FF4j ff4j(){
        FF4j ff4j = new FF4j();
        Feature papaFeature = new Feature(PAPA_FEATURE);
        papaFeature.disable(); //Default we will disable this feature later we will be enable
        ff4j.createFeature(papaFeature);
        return ff4j;
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;h5&gt;
  
  
  5. Define your API End point for Ramu Kaka and Johny to communicate.
&lt;/h5&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    @GetMapping(path = "/should-we-party")
    public String shouldWeParty(){
        if(ff4j.check(FF4jConfig.PAPA_FEATURE))
            return "Yeah Johny Lets Go for party";
        else
            return "Sorry Johny, Papa is at Home";
    }

    /*
     * End-point will change the status of the feature
     * In our case it will notify johny about status of papa
     */
    @PostMapping(path = "/papa-at-hospital/{isAtHospital}")
    public String papaAtHospital(@PathVariable boolean isAtHospital){
        if(isAtHospital){
            ff4j.enable(FF4jConfig.PAPA_FEATURE);
            return ("Message deliverd to danny: PAPA IS AT HOSPITAL, GO PARTY");
        }
        else {
            ff4j.disable(FF4jConfig.PAPA_FEATURE);
            return ("Message deliverd to danny: PAPA LEFT HOSPITAL, COME BACK");
        }
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;blockquote&gt;
&lt;p&gt;Don't forget to autowire the FF4j to instantiate before using&lt;br&gt;
&lt;/p&gt;


&lt;/blockquote&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@Autowired
    FF4j ff4j;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Results
&lt;/h3&gt;

&lt;p&gt;Following is the endpoint for &lt;strong&gt;Johny&lt;/strong&gt; to ask &lt;strong&gt;Danny&lt;/strong&gt; out.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2ftr3ogge61z60vn4xu.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2ftr3ogge61z60vn4xu.jpg" alt="should-we-party endo point" width="737" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Following is the endpoint for &lt;strong&gt;Ramu Kaka&lt;/strong&gt; to ask &lt;strong&gt;Danny&lt;/strong&gt; out.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1nhv92b3jmgy0btkbwsg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1nhv92b3jmgy0btkbwsg.png" alt="papa-at-hospital" width="713" height="367"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Once &lt;strong&gt;Ramu Kaka&lt;/strong&gt; informed &lt;strong&gt;Danny&lt;/strong&gt; that &lt;strong&gt;Big-B&lt;/strong&gt; is in hospital, &lt;strong&gt;Johny&lt;/strong&gt; will get the following response on calling &lt;strong&gt;Danny&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5imbh91fgpu96e4okut3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5imbh91fgpu96e4okut3.png" alt="Success Response" width="700" height="361"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After getting this response, Johny and Danny went for party. And the rest of the story &lt;strong&gt;cleared from browser history&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Get the full Code at &lt;a href="https://github.com/0x4E43/ff4j-sample" rel="noopener noreferrer"&gt;Github/0x4E43&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Thank you
&lt;/h2&gt;

</description>
      <category>java</category>
      <category>springboot</category>
      <category>backend</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
