<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Aidarbek</title>
    <description>The latest articles on Forem by Aidarbek (@a1darbek).</description>
    <link>https://forem.com/a1darbek</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3829883%2Ff13761c0-d77d-483e-9cd0-ba7e81b53577.jpeg</url>
      <title>Forem: Aidarbek</title>
      <link>https://forem.com/a1darbek</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/a1darbek"/>
    <language>en</language>
    <item>
      <title>What actually happens to your data when an edge node crashes</title>
      <dc:creator>Aidarbek</dc:creator>
      <pubDate>Thu, 02 Apr 2026 19:45:53 +0000</pubDate>
      <link>https://forem.com/a1darbek/what-actually-happens-to-your-data-when-an-edge-node-crashes-p2k</link>
      <guid>https://forem.com/a1darbek/what-actually-happens-to-your-data-when-an-edge-node-crashes-p2k</guid>
      <description>&lt;h2&gt;
  
  
  What actually happens to your data when an edge node crashes?
&lt;/h2&gt;

&lt;p&gt;In edge systems, failure is not an exception it’s the normal case.&lt;/p&gt;

&lt;p&gt;Power loss. Process crashes. Disk issues.&lt;/p&gt;

&lt;p&gt;Yet most pipelines are designed as if systems are stable.&lt;/p&gt;

&lt;p&gt;The real question is:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;what actually happens to your data when a node crashes mid-write?&lt;/strong&gt;&lt;/p&gt;




&lt;h3&gt;
  
  
  The reality in many edge setups
&lt;/h3&gt;

&lt;p&gt;In typical edge / IIoT pipelines, data is buffered locally:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;MQTT brokers
&lt;/li&gt;
&lt;li&gt;local files
&lt;/li&gt;
&lt;li&gt;in-memory queues
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When a crash happens, one of three things usually occurs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;partial writes
&lt;/li&gt;
&lt;li&gt;lost buffered data
&lt;/li&gt;
&lt;li&gt;manual recovery
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sometimes this is acceptable.&lt;br&gt;&lt;br&gt;
Sometimes it goes completely unnoticed until something breaks downstream.&lt;/p&gt;




&lt;h3&gt;
  
  
  What I tested instead
&lt;/h3&gt;

&lt;p&gt;Instead of focusing on throughput or benchmarks, I focused on failure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SIGKILL scenarios (no graceful shutdown)
&lt;/li&gt;
&lt;li&gt;container restarts
&lt;/li&gt;
&lt;li&gt;disk replay after crash
&lt;/li&gt;
&lt;li&gt;offset correctness under recovery
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I validated this behavior using Jepsen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result: 45/45 mixed-fault tests passed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not just happy-path performance but behavior under real failure conditions.&lt;/p&gt;




&lt;h3&gt;
  
  
  What surprised me
&lt;/h3&gt;

&lt;p&gt;The surprising part wasn’t the system behavior.&lt;/p&gt;

&lt;p&gt;It was how unclear the guarantees are in many real-world setups.&lt;/p&gt;

&lt;p&gt;In many cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;durability is assumed, not verified
&lt;/li&gt;
&lt;li&gt;recovery is “best effort”
&lt;/li&gt;
&lt;li&gt;correctness depends on implementation details
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And most teams don’t test failure scenarios explicitly.&lt;/p&gt;




&lt;h3&gt;
  
  
  The real question isn’t technical
&lt;/h3&gt;

&lt;p&gt;It’s this:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;how often does this actually matter in practice?&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Do teams really lose data during crashes?
&lt;/li&gt;
&lt;li&gt;Or is it “good enough” most of the time?
&lt;/li&gt;
&lt;li&gt;Where does it become unacceptable?
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;industrial monitoring
&lt;/li&gt;
&lt;li&gt;financial events
&lt;/li&gt;
&lt;li&gt;critical telemetry
&lt;/li&gt;
&lt;/ul&gt;




&lt;h3&gt;
  
  
  I’m trying to understand this
&lt;/h3&gt;

&lt;p&gt;I’m currently exploring this space by building and testing failure scenarios,&lt;br&gt;&lt;br&gt;
but more importantly I’m trying to map real-world experience.&lt;/p&gt;

&lt;p&gt;If you’re working on edge / IIoT systems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Have you seen data loss after crashes?
&lt;/li&gt;
&lt;li&gt;How do you recover today?
&lt;/li&gt;
&lt;li&gt;Is this a real pain or an acceptable trade-off?
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I’m not trying to sell anything just trying to understand where this actually matters.&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkt16cqnmbpvqh3axo3mc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkt16cqnmbpvqh3axo3mc.png" alt=" " width="800" height="490"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>devops</category>
      <category>iot</category>
      <category>backend</category>
    </item>
  </channel>
</rss>
