Forem: Tuntufye Mwakalasya

DISTRIBUTED SYSTEMS

Tuntufye Mwakalasya — Wed, 17 Dec 2025 16:35:35 +0000

Distributed vs Decentralized systems.

The most important distinction between the two is integrative view and expansive view
An integrative view is when there is a need to connect existing computer systems to each other.

An expansive view is when an existing system requires an extension through additional computers.

Distributed vs. Decentralized Systems

A key distinction between distributed and decentralized systems lies in their conceptual origin, which can be understood through two primary perspectives: the integrative view and the expansive view.

Core Concepts: Integrative vs. Expansive Views

Integrative View: This perspective arises from the need to connect and integrate pre-existing, often autonomous, computer systems. The goal is to make separate systems work together while respecting their administrative boundaries.
Expansive View: This perspective emerges when an existing, centrally-managed system needs to be extended or scaled out by adding more computers. The goal is to enhance the system's capabilities, such as performance or fault tolerance, while presenting a unified, single-system image to the user.

Decentralized Systems: The Integrative View

Decentralized systems are typically formed when the processes and resources of a networked system are necessarily split across multiple, often administratively separate, computers. They are born from the desire to connect existing systems that must remain independent.

Example: Federated Learning

In traditional machine learning (ML), massive datasets are brought to a central High-Performance Computing (HPC) cluster for model training. However, when data must remain within an organization's boundaries due to privacy or legal constraints, the training must be brought to the data.

Federated learning enables this by running multiple, parallel training sessions on separate, localized datasets. Each session produces a "local model." These local models are then aggregated (e.g., through model weight averaging) to build a more generalized "global model." This approach contrasts with centralized techniques where all datasets are merged into a single location for one large training session.

Distributed Systems: The Expansive View

A distributed system is a networked computer system where processes and resources are split across multiple computers to achieve scalability and reliability, while appearing to users as a single, coherent system. These systems are typically associated with the expansive view.

Example: Google Mail

Consider an email service like Gmail. A user configures their client with server addresses like imap.gmail.com and smtp.gmail.com, giving the impression of interacting with just two machines.

In reality, with billions of users, the service is supported by a massive, complex system spread across countless computers in data centers worldwide. This distributed system is designed to:

Ensure Scalability: Handle the immense load of millions of concurrent users.

Provide Fault Tolerance: Minimize the risk of losing mail due to hardware or software failures.

Maintain Transparency: Hide the underlying complexity from the end-user, who only sees a simple, unified service. The system expands or shrinks based on user demand and dependability requirements.

Key Challenges in Distributed & Decentralized Systems

Understanding the distinction between these systems is crucial because both face a unique set of complex challenges that are not present in single-machine systems.

Partial Failures: Unlike a centralized system that either works or fails completely, these systems can experience partial failures where one component fails while others continue to run. This makes error detection and recovery incredibly complex.
High Dynamism: Nodes (participating computers) can join and leave the network frequently and unpredictably. This dynamism requires sophisticated, automated management and maintenance protocols.
Security Vulnerabilities: Because these systems are networked, used by many applications, and often span multiple administrative domains, they are inherently vulnerable to a wide range of security attacks.

Perspectives for Studying Distributed Systems

To fully grasp their complexity, we study distributed systems from several different perspectives:

Architectural View: Focuses on the common organizational styles and patterns to understand how components interact and what dependencies exist between them.
Process View: Examines the different forms of processes that form the software backbone, including threads, virtualization, clients, and servers.
Communication View: Concerns the mechanisms and protocols that systems provide for exchanging data between processes.
Coordination View: Describes the fundamental coordination tasks (e.g., consensus, leader election) that happen "under the hood" to allow applications to execute correctly.
Naming View: Explores how processes, resources, and other entities are named and located. Effective naming schemes are crucial for accessing any entity in the system.
Consistency & Replication View: To achieve high performance and dependability, resources are often replicated. This view analyzes the challenges of keeping all copies of a resource consistent, especially after updates.
Fault Tolerance View: Dives into the means for masking failures and enabling recovery. This is one of the toughest aspects, as it involves numerous trade-offs, and completely masking all failures is provably impossible.
Security View: Focuses on how to ensure authorized access to resources and protect the system's integrity and confidentiality.

Core Goals of Distributed Systems

Building a distributed system is complex and should only be undertaken when necessary. The primary motivations are typically centered around two goals:

Resource Sharing: A fundamental goal is to make it easy for users and applications to access and share remote resources, which can be anything from hardware (printers, disks) to software and data.
Distribution Transparency: A key objective is to hide the complexity of the system's distribution from users and applications. The system should appear as a single, unified computing environment, masking the fact that its processes and resources are physically separated. Achieving transparency is done through middleware.

Decentralized systems is when processes and resources of a networked computer are split necessarily across multiple computers,

A distributed system is a networked computer system in which processes and resources are split sufficiently across multiple computers.

Decentralized systems are normally associated with integrative views of networked systems. They come to being because we want to connect systems yet, we may be hindered by administrative boundaries, such as in AI, there is need for massive amounts of data, normally data is brought to the HPC(High performance computers) to train the models, but when data needs to stay within the constraints of an org, we need to bring the training to the data. This is called federated learning
Federated learning is an ML approach that allows for multiple separate training sessions running in parallel to run across large boundaries, for example geographically, and aggregate the results to build a generalized model (global model) in the process. More specifically, each training session uses its own dataset and gets its own local model. Local models in different training sessions will be aggregated (for example, model weight aggregation) into a global model during the training process. This approach stands in contrast to centralized ML techniques where datasets are merged for one training session.

Distributed systems are associated with the expansive view of networked systems.

A well-known example is making use
of e-mail services, such as Google Mail. What often happens is that a user logs
into the system through a Web interface to read and send mails. More often,
however, is that users configure their personal computer (such as a laptop) to
make use of a specific mail client. To that end, they need to configure a few
settings, such as the incoming and outgoing server. In the case of Google Mail,
these are imap.gmail.com and smtp.gmail.com, respectively. Logically, it seems
as if these two servers will handle all your mail. However, with an estimate
of close to 2 billion users as of 2022, it is unlikely that only two computers
can handle all their e-mails (which was estimated to be more than 300 billion
per year, that is, some 10,000 mails per second). Behind the scenes, of course,
the entire Google Mail service has been implemented and spread across many
computers, jointly forming a distributed system. That system has been set
up to make sure that so many users can process their mails (i.e., ensures
scalability), but also that the risk of losing mail because of failures, is minimal
(i.e., the system ensures fault tolerance). To the user, however, the image of
just two servers is kept up (i.e., the distribution itself is highly transparent
to the user). The distributed system implementing an e-mail service, such
as Google Mail, typically expands (or shrinks) as dictated by dependability
requirements, in turn, dependent on the number of its users.

Why do we make the distinction between decentralized and distributed systems:

There are many often unexpected dependencies that hinder understanding the behavior of our systems, such as distributed and decentralized systems always suffer from partial failures.

Secondly in systems like this, partcipating nodes which are part of our network come and go which makes this system very dynamic. This therefore requires forms of automated management and maintenance which increases the complexity of the systems.

Lastly, the fact that these systems are networked, used by many users and applications across multiple administrative zones makes them vulnerable to security attacks. Therefore understanding their behavior and the systems as a whole requires that we understand how they can be and are secured.

We will study the systems from different perspecitves:

Architectural view, what are the common organizations, styles. This will teach us how various components of existing systems interact and depend on each other.
Process view, this is about understanding the different forms of processes that occur in distributed systems, such as threads, virtualization of hardware, processes, client, servers and so on. Processes form the software backbone of distributed systems.
Communication view, concerns the facilities that distributed systems provide to exchange data between processes.
Coordination view, what happens under the hood on top of which applications are executed. Describes the fundamental coordination tasks that need to be carried out as part of the system.
Naming view, to access processes and resources we need naming. More or so naming schemes that will lead to the process resources or whatever other type of entity that is being names.
Consistency and replication view, A critical aspect of distributed systems is that they perform well in terms of efficiency and in terms of dependability. The key instrument for both aspects is replicating resources. The only problem with replication is that updates may happen, implying that all copies of a resource need to be updated as well. It is here, that keeping up the appearance of a nondistributed system becomes challenging.
Fault tolerance view, dives into the means for masking failures and their recovery. This is the toughest perspective of understanding distributed systems. This is in part to so many trade offs and also completely masking failures and their recovery is provably impossible.
Security view, there is no nonsecured distributed system, this will allow us to focus on how to ensure authorized access to resources.

In the case of building distributed systems, just because you can build one does not mean it is necessary as they are very complex unless under certain circumstances.

Resource sharing

An important goal of a distributed system is to make it easy for users and applications to access and share remote resources. Resources can be virtually anything.
Distribution Transparency

One thing that distributed systems try to hide is that it’s processes and resources are physically connected by computers that might be very long distances between them. Meaning that it tries to make its resources and processes transparent

Inside ChatGPT: Deconstructing "Attention Is All You Need" (Part 1)

Tuntufye Mwakalasya — Fri, 21 Nov 2025 21:49:32 +0000

To understand how modern Large Language Models (LLMs) like ChatGPT work, we must first understand the architecture that changed everything: the Transformer. Before we dive into the complex layers, we need to establish why we moved away from previous methods and how the model initially processes language.

1. The Predecessor: Recurrent Neural Networks (RNNs) and Their Limitations

Before the "Attention Is All You Need" paper, the standard for processing sequential data (like text) was the Recurrent Neural Network (RNN).In an RNN, data is processed sequentially. We give the network an initial state (State 0) along with an input x1 to produce an output y1 and a hidden state. This hidden state is passed forward to the next step, allowing the network to "remember" previous inputs.

The Vanishing Gradient Problem

While intuitive, RNNs suffer from severe limitations, specifically slow computation for long sequences and the vanishing or exploding gradient problem.

To understand this, let's look at calculus, specifically the Chain Rule.

If we have a composite function

\circ g)(x)

, the derivative is:

\cdot g'(x)

In a deep neural network, backpropagation involves multiplying gradients layer by layer (like the chain rule). If we have many layers, we are essentially multiplying many numbers together.

Imagine multiplying fractions like:

\frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} \times \cdots

As the number of layers (or time steps) increases, this number becomes infinitesimally small ("vanishes") or massively large ("explodes"). This makes it incredibly difficult for the model to access or learn from information that appeared early in a long sequence.

2. The Transformer Architecture

The Transformer abandons recurrence entirely, relying instead on an Encoder-Decoder architecture. It processes the entire sequence at once, which solves the speed and long-term dependency issues of RNNs.

The Input Matrix

Let's look at how data enters the model.

If we have an input sentence of length 6 (Sequence Length) and a model dimension (

d_{model}

) of 512, our input is a matrix of size

(6, 512)

Each row represents a word, and the columns (length 512) represent that word as a vector. You might ask: Why 512 dimensions?

We need high-dimensional space to capture:

Semantic Meaning: What the word actually means.
Syntactic Role: Is it a noun, verb, or adjective?
Relationships: How it relates to other words (e.g., "King" vs "Queen").
Context: Multiple contexts the word can appear in.

Input Embedding

Computers don't understand strings; they understand numbers. We take our original sentence:

"Your cat is a lovely cat"

First, we map these to Input IDs (their position in the vocabulary):
We then map these IDs into a vector of size 512. Note that these vectors are not fixed; they are learned parameters that change during training to better represent the word's meaning.

3. Positional Encoding

Since the Transformer processes words in parallel (not sequentially like an RNN), it has no inherent concept of "order." It doesn't know that "Your" comes before "cat." We must inject this information manually using Positional Encodings.

We want the model to treat words that appear close to each other as "close" mathematically. To do this, we use trigonometric functions because they naturally represent continuous patterns that the model can easily learn to extrapolate.

We add this positional vector to our embedding vector. The formula used in the paper is:

For even positions ( $2 i$ ):

\sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)

For odd positions ( $2 i + 1$ ):

\cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)

This ensures that every position has a unique encoding that is consistent across training and inference.

4. Self-Attention: The Core Mechanism

This is the "magic" of the architecture. Self-attention allows the model to relate words to each other within the same sentence. It determines how much "focus" the word "lovely" should have on the word "cat."

The formula for Scaled Dot-Product Attention is:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Where:

Q (Query): What I am looking for.
K (Key): What I contain.
V (Value): The actual content I will pass along.

The Matrix Math

For a sequence length of 6 and dimension 512:

We multiply Q (6 × 512) by K^T (512 × 6).
This results in a (6 × 6) matrix.
We apply the Softmax function. This turns the scores into probabilities (summing up to 1).

This (6 × 6) matrix captures the interaction between every word and every other word. When we multiply this by V, we get a weighted sum of the values, where the weights are determined by the compatibility of the Query and Key.

Key Benefits of Self-Attention:

Permutation Invariant: It treats the sequence as a set of relationships rather than a strict list.
Parameter Efficiency: Pure self-attention requires no learnable parameters (though the linear layers surrounding it do).
Long-range Dependencies: Words at the start of a sentence can attend to words at the end just as easily as adjacent words.

Key Benefits of Self-Attention:

Permutation Invariant: It treats the sequence as a set of relationships rather than a strict list.
Parameter Efficiency: Pure self-attention requires no learnable parameters (though the linear layers surrounding it do).
Long-range Dependencies: Words at the start of a sentence can attend to words at the end just as easily as adjacent words.

Summary & Looking Ahead

We have successfully moved away from the sequential limitations of RNNs and embraced the parallel nature of Transformers. We've learned how to convert text into meaningful vector spaces, inject order using positional encoding, and, most importantly, derive the mathematical foundation of how words "pay attention" to each other using Queries, Keys, and Values.

But there is a catch.

The mechanism we just described, a single pass of

softmax(QKTdk)V\text{softmax}(\frac{QK^T}{\sqrt{d_k}})V

, is only capable of focusing on one type of relationship at a time. For example, it might focus heavily on syntactic relationships (such as subject-verb agreement) but completely miss semantic nuances (like sarcasm or references).

Real-world language is too complex for a single "gaze." To build a model like ChatGPT, we need it to look at the sentence through multiple lenses simultaneously.

In Part 2, we will take the self-attention mechanism and clone it, creating Multi-Head Attention. We will then see how these attention scores are processed through Feed-Forward Networks to finally construct the complete Transformer block.

Stay tuned.

The Two Lists That Define Every Software Project

Tuntufye Mwakalasya — Fri, 14 Nov 2025 21:43:22 +0000

If you’ve ever been near a software developer, you’ve probably heard a frustrated groan followed by the classic phrase: "But it worked on my machine!"

This, and a million other frustrations like File Not Found or Symbol Not Found, often boil down to one of the most misunderstood parts of software engineering. It’s not a bug in the code, but a problem with the lists.

The problem is that a computer is not a mind reader. It’s an incredibly fast, precise, and literal-minded robot. To get it to build your software, you have to give it two separate things: a Recipe and a Shopping List.

And the central conflict of all software development is that the robot never reads the Recipe to figure out the Shopping List.

👨‍🍳 The Metaphor: The Robot Chef

Imagine you have a robot chef. Its job is to bake a cake.

The Recipe: This is your source code. It's the "ground truth" of what needs to be done. It might say "Step 1: Mix flour, eggs, and sugar."
The Shopping List: This is your build file (a Makefile, BUILD.bazel, package.json, etc.). It's the list of ingredients you claim are needed.
The Robot Chef: This is your build tool (like make, Bazel, or npm).

The robot's process is simple and unforgiving:

It reads only your Shopping List.
It goes to the store and gathers every single item on that list.
It returns to the kitchen and tries to follow the Recipe using only the items it just bought.

This simple process can fail in two major ways.

Scenario 1: The Broken Build (The Missing Ingredient)

This is the most common error.

The Shopping List: You write "flour, sugar."
The Robot's Action: The robot fetches flour and sugar.
The Recipe (Code): "Step 1: Mix flour, eggs, and sugar."
The Result: FAILURE.

The robot stops, drops the bowl, and reports FATAL ERROR: Ingredient 'eggs' not found. It doesn't matter that "eggs" are obviously needed. They weren't on the list.

This is a Missing Dependency. In technical terms, the Actual Dependency Graph (we'll call it Ga) included an edge from Cake to Eggs, but the Declared Dependency Graph (Gd) did not.

The build fails because your declared list was not a perfect representation of reality. You told the build tool a lie, and the compiler caught it.

[Your diagram/image for Scenario 1 here]

Scenario 2: The Slow Build (The Useless Ingredient)

This is a more subtle but equally important problem.

The Shopping List: You write "flour, sugar, eggs... and cabbage."
The Robot's Action: The robot fetches flour, sugar, eggs, and a head of cabbage.
The Recipe (Code): "Step 1: Mix flour, eggs, and sugar..." The cake bakes perfectly.
The Result: SUCCESS... but.

The build succeeded. The cake is delicious. But you now have a head of cabbage rotting on the counter, and the robot's shopping trip took twice as long.

This is Overapproximation. Your build is "correct" because the golden rule (Ga is a subset of Gd) is met. But it's inefficient. You've introduced build bloat. The build tool wasted time and resources compiling, linking, and processing a library (cabbage) that was never used. In a large project, this is the difference between a 2-minute build and a 40-minute build.

Scenario 3: The "It Works On My Machine" Nightmare

This is the most complex problem, and it’s where our metaphor gets really useful.

Let's say you're making two things: a Chocolate Cake and Brownies.

Chocolate Cake: Your Shopping List correctly lists "Cake Mix." This "Cake Mix" happens to include a bag of "Cocoa Powder" inside.
Brownies: Your Recipe for brownies actually needs "Cocoa Powder," but you forget to put it on the Brownie Shopping List.

You build the Chocolate Cake first. The robot buys the "Cake Mix" and leaves the "Cocoa Powder" on the counter.
Then you build the Brownies. The robot (which should fail) looks at the empty Brownie Shopping List, but then sees the leftover "Cocoa Powder" on the counter from the last build. It shrugs, uses it, and the build succeeds!

Then, the disaster: Your co-worker, trying to be efficient, switches the Chocolate Cake recipe to "Vanilla Cake Mix."

Suddenly, your Brownie build breaks. You're staring at your screen, shouting "But I didn't even touch the Brownie code!"

You were relying on a "ghost." This is a Transitive Dependency nightmare.

Your Brownie_App (Recipe) had an actual dependency on Cocoa_Library.
You never declared it.
It only worked because you declared a dependency on Cake_Mix_Library, which transitively depended on Cocoa_Library.

The moment Cake_Mix_Library no longer needed Cocoa_Library, your build failed. Modern build systems like Bazel are designed to prevent this. They enforce strict dependency checking, essentially "cleaning the counter" between every single step to ensure you're not using ingredients you didn't explicitly ask for.

The Takeaway

Writing software isn't just about the Recipe (code). It's about meticulously maintaining the Shopping List (build file).

If you miss an item, your build breaks.
If you add extra items, your build slows down.
If you "borrow" items from another recipe's list, your build becomes a fragile, ticking time bomb.

A good developer is a good chef. A great developer writes a perfect shopping list every single time.

Building a Mini Build System in Go: Understanding How Bazel Works Under the Hood

Tuntufye Mwakalasya — Sat, 08 Nov 2025 00:27:40 +0000

Imagine you're running a restaurant kitchen. You have recipes (build targets), ingredients (source files), and some dishes that need other dishes to be ready first (dependencies). How do you organize this chaos? That's exactly what build systems like Bazel, Make, and our Mini-Bazel do for software.

Why Build Systems Matter (The Busy Kitchen Problem)

Picture this: You're making a sandwich. You need:

Bread (must be toasted first)
Butter (must be softened)
Cheese (must be sliced)

If the bread is already toasted from earlier, why toast it again?
If someone changed the cheese type, you need to remake the sandwich.
This is exactly what build systems figure out for code!

In our Mini-Bazel example:

my_app is like the final sandwich
utils.a is like the toasted bread (a prerequisite)
The .go files are our raw ingredients // Let's break down our BUILD.mini file for everyone:

/*
Think of this as a recipe book where each recipe has:
- A name (what we're making)
- Instructions (the command to run)  
- Ingredients (source files)
- Other dishes needed first (dependencies)
*/

- name: "utils.a"          # Like "Toasted Bread"
  cmd: "go build..."       # "Put bread in toaster for 2 min"
  srcs: ["utils/greet.go"] # "You need: sliced bread"
  deps: []                 # "No other dishes needed first"

- name: "my_app"           # Like "Complete Sandwich"
  cmd: "go build..."       # "Assemble all parts"
  srcs: ["main.go"]        # "You need: plate, lettuce"
  deps: ["utils.a"]        # "First make: toasted bread"

The Two Big Questions Every Build System Answers

Remember our kitchen? Every build system (chef) must answer:

Question 1: "What order should I cook things?" (The Scheduler)

Option A - The Prep Cook (Topological/Make-style):
"I'll list everything needed for dinner, sort them by dependencies,
then cook in that exact order."

Pro: Simple and predictable
Con: Can't handle surprises (dynamic dependencies)

Option B - The Adaptive Chef (Restarting/Excel-style):
"I'll start cooking, and if I realize I need something not ready,
I'll switch to making that first."

Pro: Handles surprises well
Con: Might restart dishes multiple times

Option C - The Multitasker (Suspending/Shake-style):
"I'll start multiple dishes, pause any that need something else,
and resume when ready."

Pro: Very efficient
Con: More complex to implement

Question 2: "How do I know if something needs remaking?" (The Rebuilder)

Check the Timer (Timestamp/Make-style):
"If ingredients arrived after the dish was made, remake it."

Taste Test (Verifying Traces/Shake-style):
"I remember exactly what went into this dish. If any of those
ingredients changed, remake it."

Recipe Card System (Constructive Traces/Bazel-style):
"I keep cards saying 'these exact ingredients make this exact dish'.
If someone else made it already, just use theirs!"

Here's how our Mini-Bazel starts (Part 1: Loading recipes)

func main() {
    // Step 1: Read the recipe book
    file, _ := os.ReadFile("BUILD.mini")

    // Step 2: Understand what each recipe means
    var targets []Target  // Our list of recipes
    yaml.Unmarshal(file, &targets)  // Parse the recipes

    // Step 3: Now we know what we CAN cook
    // (Part 2 will figure out what we SHOULD cook)
}

Coming in Part 2: The Cooking Phase

Now that our Mini-Bazel can read recipes, we need to teach it to:

Build a dependency graph (figure out cooking order)
Detect what needs rebuilding (check if ingredients changed)
Execute builds in the right order (actually cook!)

We'll implement a simple scheduler (topological sort) and a basic
rebuilder (timestamp checking), creating a Make-like build system
in Go!

Understanding the CAP Theorem Through a Hands-On Simulation in Golang

Tuntufye Mwakalasya — Mon, 16 Dec 2024 01:29:34 +0000

Introduction
Distributed systems are the backbone of modern software, enabling scalability and fault tolerance across networks. However, designing such systems comes with challenges, especially when ensuring reliability during failures. One fundamental principle in distributed systems is the CAP Theorem, which highlights the trade-offs every system must make.

In this blog, we’ll explore the CAP Theorem through a practical simulation in Golang, showcasing how consistency, availability, and partition tolerance interact in real-world systems.

What is the CAP Theorem?
The CAP Theorem, introduced by Eric Brewer in 2000, states that a distributed system can only guarantee two out of the following three properties:

Consistency (C):

All nodes in the system return the same data at the same time.
Example: When you update a database, all replicas immediately reflect the update.

Availability (A):

Every request receives a response, even during failures.
Example: A load balancer that always returns a result, even if it’s stale.

Partition Tolerance (P):

The system continues operating despite network partitions.
Example: Nodes can still process requests independently if communication between them is lost.
Key Insight: During a network partition, a system must sacrifice either Consistency or Availability, but not both.

Building the Simulation
To understand CAP trade-offs, we will build a simple simulation in Golang. The system consists of:

Nodes: Represent individual components of the system, each with a counter.
Cluster: Manages the nodes and synchronizes their state.

Node Struct
Each node has:
A name for identification.
A counter to hold data.
A partitioned flag to indicate if the node is disconnected.

type Node struct {
    name        string
    counter     int
    partitioned bool
}

Cluster Struct
The cluster organizes multiple nodes and provides synchronization:

type Cluster struct {
    nodes []*Node
}

Core Methods

1. Write Operation
The Write method updates a node’s counter only if it is not partitioned:

func (n *Node) Write(value int) {
    if !n.partitioned {
        n.counter = value
        fmt.Printf("Node %s updated to %d\n", n.name, n.counter)
    } else {
        fmt.Printf("Node %s is partitioned, Write failed\n", n.name)
    }
}

Synchronization The syncNodes method propagates updates from one node to others in the cluster: Skips the updated node. Skips nodes that are partitioned.

func (c *Cluster) syncNodes(updatedNode *Node) {
    for _, node := range c.nodes {
        if node.name == updatedNode.name || node.partitioned {
            continue
        }
        node.counter = updatedNode.counter
        fmt.Printf("Node %s synchronized to %d\n", node.name, updatedNode.counter)
    }
}

Simulating CAP Trade-Offs
Scenario 1: Consistency
When all nodes are connected, synchronization ensures that every node has the same data:

nodeA.Write(10)
cluster.syncNodes(nodeA)

Output:

Node Node A updated to 10
Node Node B synchronized to 10
Node Node C synchronized to 10
Node Node D synchronized to 10

Scenario 2: Partition Tolerance
When a node is partitioned, it cannot participate in synchronization:

nodeB.partitioned = true
nodeA.Write(20)
cluster.syncNodes(nodeA)

Output:

Node Node A updated to 20
Node Node B is partitioned, sync skipped
Node Node C synchronized to 20
Node Node D synchronized to 20

Scenario 3: Availability
Despite partitions, nodes continue operating independently. Writes to partitioned nodes may result in stale data:

nodeB.Write(30)

Output:


Node Node B is partitioned, Write failed

Lessons Learned

From this simulation, we observed:
Consistency vs. Availability: Synchronization ensures consistency but sacrifices availability for partitioned nodes.
Partition Tolerance: Partitioned nodes remain functional but risk divergence in state.
Trade-Offs Are Inevitable: Designing distributed systems requires clear prioritization based on the use case.

Next Steps
This simulation is just the beginning. Potential extensions include:

Asynchronous Synchronization: Use Goroutines to simulate real-world latencies.
Recovery Mechanisms: Handle nodes rejoining the cluster after partitions.
Monitoring: Add metrics for latency, synchronization rates, and failure handling.

Conclusion
The CAP Theorem encapsulates the complexity of distributed systems. Through this Golang simulation, we gained hands-on experience with its principles and trade-offs. Whether building databases or scalable services, understanding CAP is key to making informed architectural decisions.

What are your thoughts on the CAP Theorem? Have you faced similar trade-offs in your projects? Let me know in the comments!