Forem: Javad

Advanced GPU Optimization: Metal & Vulkan Compute from zero to hero - Part 2

Javad — Sat, 14 Feb 2026 16:41:09 +0000

Welcome back, GPU programming experts! You've mastered the fundamentals of Metal and Vulkan compute. Now it's time to explore the advanced features that truly unlock the power of modern GPUs. In this comprehensive guide, we'll dive deep into:

Metal Performance Shaders (MPS) – Apple's library of highly optimized compute primitives.
Metal Ray Tracing – Hardware‑accelerated ray tracing on Apple Silicon.
Vulkan Subgroup Operations – Efficient intra‑workgroup communication.
Vulkan Ray Tracing – Cross‑platform ray tracing extensions.
Vulkan Video – Hardware‑accelerated video encoding/decoding.
Cross‑Platform Alternatives – How SYCL and OpenCL fit into the picture.

By the end, you'll be equipped to build cutting‑edge applications that leverage the full capabilities of modern GPUs.

Part 1: Metal Performance Shaders (MPS)

Metal Performance Shaders is a framework that provides a collection of highly optimized compute and graphics functions for image processing, machine learning, linear algebra, and more. MPS is tightly integrated with Metal and takes advantage of the specific hardware capabilities of Apple GPUs.

1.1 Overview and Setup

MPS is available on all Apple platforms (iOS, macOS, tvOS) with Metal support. To use MPS, import the framework:

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>

MPS kernels are objects that you create with a specific device and then encode into a Metal command buffer. They manage their own internal state and resources, so you don't have to worry about shader compilation or pipeline state.

1.2 Key MPS Features

Image Processing: Convolution, morphology, histogram, resizing, etc.
Matrix Multiplication: MPSMatrixMultiplication for fast linear algebra.
Neural Network: MPSCNNConvolution, MPSCNNFullyConnected, MPSCNNPooling – building blocks for deep learning.
Ray Tracing: MPSRayIntersector for ray‑triangle intersection and acceleration structure building.
Random Number Generation: MPSParallelRandom for GPU‑side random numbers.

1.3 Example: Matrix Multiplication with MPS

Let's see how to perform matrix multiplication using MPSMatrixMultiplication. This is far simpler than writing your own kernel and often faster.

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>

id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];

// Create matrices as MPSMatrix objects (wrappers around MTLBuffer)
MPSMatrixDescriptor *desc = [MPSMatrixDescriptor matrixDescriptorWithRows:1024 columns:1024 rowBytes:1024 * sizeof(float) dataType:MPSDataTypeFloat32];
id<MTLBuffer> bufferA = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferB = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferC = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];

MPSMatrix *matrixA = [[MPSMatrix alloc] initWithBuffer:bufferA descriptor:desc];
MPSMatrix *matrixB = [[MPSMatrix alloc] initWithBuffer:bufferB descriptor:desc];
MPSMatrix *matrixC = [[MPSMatrix alloc] initWithBuffer:bufferC descriptor:desc];

// Create the multiplication kernel
MPSMatrixMultiplication *matMul = [[MPSMatrixMultiplication alloc] initWithDevice:device transposeLeft:NO transposeRight:NO resultRows:1024 resultColumns:1024 interiorColumns:1024 alpha:1.0 beta:0.0];

// Encode the kernel
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
[matMul encodeToCommandBuffer:commandBuffer leftMatrix:matrixA rightMatrix:matrixB resultMatrix:matrixC];
[commandBuffer commit];
[commandBuffer waitUntilCompleted];

// Result in bufferC

That's it! The kernel automatically chooses the optimal algorithm for the device.

1.4 Example: Convolution for Deep Learning

MPS provides a full suite of neural network layers. Here's a simple convolution layer:

MPSCNNConvolutionDescriptor *convDesc = [MPSCNNConvolutionDescriptor cnnConvolutionDescriptorWithKernelWidth:3 kernelHeight:3 inputFeatureChannels:3 outputFeatureChannels:64];
convDesc.strideInPixelsX = 1; convDesc.strideInPixelsY = 1;

MPSCNNConvolution *conv = [[MPSCNNConvolution alloc] initWithDevice:device convolutionDescriptor:convDesc kernelWeights:weightsData biasTerms:biasData flags:MPSCNNConvolutionFlagsNone];

// Assume you have an MPSImage *srcImage
[conv encodeToCommandBuffer:commandBuffer sourceImage:srcImage destinationImage:dstImage];

MPS automatically handles weight layout and uses hardware‑accelerated matrix multiplication or Winograd algorithms.

1.5 MPSRayIntersector for Ray Tracing

MPSRayIntersector provides a fast, high‑level interface for ray tracing on Apple GPUs. It builds acceleration structures (BVHs) and performs ray‑primitive intersection tests.

Basic usage:

Create MPSRayIntersector with a device.
Build an acceleration structure from vertex and index data.
Encode intersection tests with rays.

MPSRayIntersector *intersector = [[MPSRayIntersector alloc] initWithDevice:device];

// Create a vertex buffer and index buffer
id<MTLBuffer> vertexBuffer = [device newBufferWithBytes:vertices length:vertexDataSize options:MTLResourceStorageModeManaged];
id<MTLBuffer> indexBuffer = [device newBufferWithBytes:indices length:indexDataSize options:MTLResourceStorageModeManaged];

// Build acceleration structure
MPSPolygonBuffer *polygonBuffer = [MPSPolygonBuffer polygonBufferWithDevice:device];
polygonBuffer.vertexBuffer = vertexBuffer;
polygonBuffer.indexBuffer = indexBuffer;
NSArray<MPSPolygonBuffer *> *buffers = @[polygonBuffer];
id<MTLBuffer> vertexCountBuffer = ...; // optional per‑polygon buffer

MPSTriangleAccelerationStructure *accel = [[MPSTriangleAccelerationStructure alloc] initWithDevice:device];
accel.polygonBuffers = buffers;
[accel rebuild];

// Prepare ray buffer (with origin, direction, etc.)
id<MTLBuffer> rayBuffer = [device newBufferWithLength:rayCount * sizeof(MPSRayOriginMaskDirection) options:MTLResourceStorageModeManaged];
MPSRayOriginMaskDirection *rays = (MPSRayOriginMaskDirection*)[rayBuffer contents];
// fill rays...

// Encode intersection
MPSIntersectionDataType intersectionDataType = MPSIntersectionDataTypeDistance;
id<MTLBuffer> intersectionBuffer = [device newBufferWithLength:rayCount * sizeof(MPSIntersectionDistance) options:MTLResourceStorageModeManaged];
[intersector encodeIntersectionToCommandBuffer:commandBuffer intersectionType:MPSIntersectionTypeNearest rayBuffer:rayBuffer rayBufferOffset:0 intersectionBuffer:intersectionBuffer intersectionBufferOffset:0 rayCount:rayCount accelerationStructure:accel];

You can then read the intersection distances from intersectionBuffer. MPSRayIntersector supports both triangle and sphere primitives, and can handle large scenes efficiently.

1.6 Performance Tips for MPS

Reuse MPS kernels and acceleration structures across frames.
Use MPSKernelOptions to tune for performance (e.g., MPSKernelOptionsAllowReducedPrecision for FP16).
For neural networks, consider using MPSNNGraph to fuse multiple layers into a single optimized graph.
Profile with Xcode's Metal System Trace to see how MPS kernels are scheduled.

Part 2: Metal Ray Tracing

Beyond MPSRayIntersector, Metal provides a more explicit ray tracing API that integrates with the Metal shading pipeline. This allows you to write custom ray generation, any‑hit, closest‑hit, and miss shaders.

2.1 Metal Ray Tracing Concepts

Acceleration Structure: A BVH built from geometry (primitives).
Intersection Function: A shader that determines if a ray hits a primitive. Metal provides a default triangle intersection, but you can write custom ones.
Ray Tracing Pipeline: Similar to a compute pipeline, but specialized for ray tracing.
Shader Stages: raygen, intersection, anyhit, closesthit, miss.

2.2 Setting Up a Basic Ray Tracer

Step 1: Create an acceleration structure
Similar to MPS, but using Metal's MTLAccelerationStructure APIs. You build a primitive acceleration structure descriptor, then encode the build into a command buffer.

Step 2: Create a ray tracing pipeline
You need a MTLRayTracingPipelineDescriptor that specifies the shader functions.

Step 3: Allocate resources

Ray buffer (origin, direction, etc.)
Intersection result buffer
Texture for output

Step 4: Encode ray tracing work
Use MTLRayTracingCommandEncoder to set the pipeline, bind resources, and dispatch rays.

2.3 Example: Simple Ray Generation Shader (MSL)

#include <metal_stdlib>
using namespace metal;

struct Ray {
    float3 origin;
    float3 direction;
};

struct Intersection {
    float distance;
    uint primitiveIndex;
    float2 barycentricCoord;
};

kernel void raygen(device Ray* rays [[buffer(0)]],
                   device Intersection* intersections [[buffer(1)]],
                   texture2d<float, access::write> output [[texture(0)]],
                   uint2 tid [[thread_position_in_grid]]) {
    // Generate ray for pixel (tid.x, tid.y)
    float2 uv = float2(tid) / float2(output.get_width(), output.get_height());
    rays[tid.y * output.get_width() + tid.x] = Ray{ float3(uv * 2.0f - 1.0f, 0.0f), float3(0,0,1) };

    // Wait for intersection results (simplified)
    Intersection hit = intersections[tid.y * output.get_width() + tid.x];
    float3 color = (hit.distance < 1e30) ? float3(1,0,0) : float3(0,0,0);
    output.write(float4(color, 1), tid);
}

You'll also need a closest‑hit shader that computes color based on material.

2.4 Building the Pipeline

On the host side, you compile the shaders and create a pipeline:

id<MTLLibrary> library = [device newDefaultLibrary];
id<MTLFunction> raygenFunc = [library newFunctionWithName:@"raygen"];
id<MTLFunction> closestHitFunc = [library newFunctionWithName:@"closesthit"];
id<MTLFunction> missFunc = [library newFunctionWithName:@"miss"];

MTLRayTracingPipelineDescriptor *desc = [MTLRayTracingPipelineDescriptor new];
desc.rayGeneratorFunction = raygenFunc;
desc.closestHitFunction = closestHitFunc;
desc.missFunction = missFunc;
desc.maxCallStackDepth = 1; // for recursion
desc.maxPayloadSize = sizeof(float3); // per‑ray data
desc.maxAttributeSize = sizeof(float2); // barycentrics

id<MTLRayTracingPipelineState> pipeline = [device newRayTracingPipelineStateWithDescriptor:desc error:nil];

2.5 Encoding and Dispatch

id<MTLRayTracingCommandEncoder> rtEncoder = [commandBuffer rayTracingCommandEncoder];
[rtEncoder setRayTracingPipelineState:pipeline];
[rtEncoder setRayBuffer:rayBuffer offset:0 atIndex:0];
[rtEncoder setIntersectionBuffer:intersectionBuffer offset:0 atIndex:1];
[rtEncoder setTexture:outputTexture atIndex:0];

// Dispatch rays (one per pixel)
[rtEncoder dispatchRays:MTLSizeMake(width, height, 1)];
[rtEncoder endEncoding];

This is a minimal example. Full ray tracing requires handling acceleration structures, intersection queries, and multiple hit groups.

2.6 Performance Considerations

Use MTLAccelerationStructure for static geometry; rebuild only when needed.
Batch rays in coherent groups for better coherence.
Use the smallest possible payload and attribute sizes.
Profile with Xcode to identify bottlenecks.

Part 3: Vulkan Subgroup Operations

Subgroup operations are a powerful feature in Vulkan (and other APIs) that allow efficient communication and computation among a set of invocations that execute together (a subgroup, roughly corresponding to a CUDA warp or AMD wavefront). They can dramatically improve performance for reductions, scans, and other data‑parallel patterns.

3.1 What are Subgroups?

A subgroup is a collection of invocations within a workgroup that can communicate and synchronize using specialized operations. The size of a subgroup is implementation‑defined but can be queried. On NVIDIA, it's 32; on AMD, 64; on Intel, maybe 8,16,32.

Subgroup operations include:

Vote: subgroupAll(), subgroupAny(), subgroupBallot().
Broadcast: subgroupBroadcast().
Shuffle: subgroupShuffle(), subgroupShuffleXor().
Arithmetic: subgroupAdd(), subgroupMin(), etc. (reductions).
Scan: subgroupInclusiveAdd(), subgroupExclusiveAdd().

3.2 Enabling Subgroup Support in Vulkan

You need to check for the VK_EXT_subgroup_size_control extension or at least the core subgroup support (since Vulkan 1.1, subgroups are core but optional). In your shader, declare:

#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable
#extension GL_KHR_shader_subgroup_ballot : enable
#extension GL_KHR_shader_subgroup_shuffle : enable

3.3 Example: Parallel Reduction with Subgroups

Suppose you want to sum an array within a workgroup. Without subgroups, you'd use shared memory and barriers. With subgroups, you can do a hierarchical reduction:

#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable

layout(local_size_x = 256) in;
layout(binding = 0) buffer In { float data[]; } inBuf;
layout(binding = 1) buffer Out { float data[]; } outBuf;

void main() {
    uint gid = gl_GlobalInvocationID.x;
    uint lid = gl_LocalInvocationID.x;
    float val = inBuf.data[gid];

    // First reduce within subgroup
    float subgroupSum = subgroupAdd(val);

    // The first invocation in each subgroup writes to shared memory
    shared float sharedSums[gl_WorkGroupSize.x / gl_SubgroupSize];
    if (subgroupElect()) {
        sharedSums[lid / gl_SubgroupSize] = subgroupSum;
    }
    barrier();

    // One subgroup reduces the shared sums
    if (gl_SubgroupID == 0) {
        float total = (lid < gl_NumSubgroups) ? sharedSums[lid] : 0;
        total = subgroupAdd(total);
        if (subgroupElect() && lid == 0) {
            outBuf.data[gl_WorkGroupID.x] = total;
        }
    }
}

This is much faster than a naive loop because it leverages hardware acceleration.

3.4 Vulkan Subgroup Query on Host

You can query subgroup properties:

VkPhysicalDeviceSubgroupProperties subgroupProps = {};
subgroupProps.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SUBGROUP_PROPERTIES;
VkPhysicalDeviceProperties2 props2 = {};
props2.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2;
props2.pNext = &subgroupProps;
vkGetPhysicalDeviceProperties2(physicalDevice, &props2);

uint32_t subgroupSize = subgroupProps.subgroupSize;
// Check supported operations in subgroupProps.supportedOperations

3.5 Best Practices

Use subgroups when you need intra‑workgroup communication with low latency.
Avoid branching that diverges within a subgroup (all invocations in a subgroup should take the same path).
Prefer subgroup operations over shared memory for small reductions/scans.
Fall back to shared memory if subgroup operations are not supported (query via gl_SubgroupSize == 0 or use #ifdef).

Part 4: Vulkan Ray Tracing

Vulkan ray tracing extensions (VK_KHR_ray_tracing_pipeline, VK_KHR_acceleration_structure, etc.) provide a cross‑platform API for hardware‑accelerated ray tracing. They are supported on NVIDIA RTX, AMD RDNA2+, and Intel Arc GPUs.

4.1 Core Concepts

Acceleration Structure: A BVH built from bottom‑level (geometry) and top‑level (instances) structures.
Shader Binding Table (SBT): A table of shader handles for ray generation, hit groups, and miss shaders.
Ray Tracing Pipeline: Combines shader stages (raygen, miss, closesthit, anyhit, intersection).
Ray Tracing Commands: vkCmdTraceRaysKHR to launch rays.

4.2 Building Acceleration Structures

First, create buffers with geometry data. Then use vkCmdBuildAccelerationStructuresKHR to build bottom‑level and top‑level AS.

Example: Build a bottom‑level AS from a triangle mesh:

VkAccelerationStructureGeometryKHR geometry = {};
geometry.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_KHR;
geometry.geometryType = VK_GEOMETRY_TYPE_TRIANGLES_KHR;
geometry.flags = VK_GEOMETRY_OPAQUE_BIT_KHR;
geometry.geometry.triangles.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_TRIANGLES_DATA_KHR;
geometry.geometry.triangles.vertexFormat = VK_FORMAT_R32G32B32_SFLOAT;
geometry.geometry.triangles.vertexData.deviceAddress = vertexBufferAddress;
geometry.geometry.triangles.vertexStride = sizeof(Vertex);
geometry.geometry.triangles.maxVertex = vertexCount - 1;
geometry.geometry.triangles.indexType = VK_INDEX_TYPE_UINT32;
geometry.geometry.triangles.indexData.deviceAddress = indexBufferAddress;

VkAccelerationStructureBuildRangeInfoKHR range = {};
range.primitiveCount = indexCount / 3;
range.primitiveOffset = 0;
range.firstVertex = 0;
range.transformOffset = 0;

// Create and build bottom AS...

You'll need to query the required memory size, allocate, and then build.

4.3 Shader Example (GLSL)

Ray generation shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(binding = 0) uniform accelerationStructureEXT topLevelAS;
layout(binding = 1, rgba8) uniform image2D image;
layout(location = 0) rayPayloadEXT vec3 hitColor;

void main() {
    ivec2 pixel = ivec2(gl_LaunchIDEXT.xy);
    vec2 dim = vec2(gl_LaunchSizeEXT.xy);
    vec2 d = (vec2(pixel) + vec2(0.5)) / dim - 0.5;
    float aspect = dim.x / dim.y;
    vec3 origin = vec3(0,0,-2);
    vec3 direction = normalize(vec3(d.x * aspect, -d.y, 1));

    traceRayEXT(topLevelAS, gl_RayFlagsOpaqueEXT, 0xff, 0, 0, 0, origin, 0.001, direction, 1000.0, 0);
    imageStore(image, pixel, vec4(hitColor, 1));
}

Closest hit shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(location = 0) rayPayloadInEXT vec3 hitColor;
hitAttributeEXT vec2 attribs;

void main() {
    hitColor = vec3(1,0,0); // red
}

Miss shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(location = 0) rayPayloadInEXT vec3 hitColor;

void main() {
    hitColor = vec3(0,0,0); // black
}

4.4 Creating a Ray Tracing Pipeline

You need to create shader modules for each stage, then a ray tracing pipeline with VkRayTracingPipelineCreateInfoKHR. This includes the shader groups (raygen, miss, hit groups).

4.5 Shader Binding Table (SBT)

The SBT is a buffer that holds shader handles and any local data for each shader group. You must fill it according to the pipeline layout. It's complex but crucial.

4.6 Dispatching Rays

After recording the SBT buffer, use:

vkCmdTraceRaysKHR(commandBuffer, &raygenSBT, &missSBT, &hitSBT, &callableSBT, width, height, 1);

4.7 Performance Tips

Use opaque geometry and avoid any‑hit shaders when possible.
Keep ray payload small.
Group rays coherently (e.g., sort by direction).
Use inline ray tracing (VK_KHR_ray_query) for simpler tasks without full pipeline overhead.

Part 5: Vulkan Video

Vulkan Video extensions (VK_KHR_video_queue, VK_KHR_video_decode_queue, VK_KHR_video_encode_queue) provide hardware‑accelerated video encoding and decoding. This is a relatively new area but powerful for applications needing real‑time video processing.

5.1 Overview

Vulkan Video exposes video codec capabilities as Vulkan queues and operations. You can decode H.264, H.265, etc., directly to Vulkan images, and encode from images. It integrates seamlessly with the rest of Vulkan for post‑processing and rendering.

5.2 Setting Up Video Queues

When creating a logical device, you need to request a video decode/encode queue:

float priority = 1.0f;
VkDeviceQueueCreateInfo queueCreateInfo = {};
queueCreateInfo.sType = VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO;
queueCreateInfo.queueFamilyIndex = videoQueueFamilyIndex;
queueCreateInfo.queueCount = 1;
queueCreateInfo.pQueuePriorities = &priority;

You must first query queue families that support VK_QUEUE_VIDEO_DECODE_BIT_KHR or VK_QUEUE_VIDEO_ENCODE_BIT_KHR.

5.3 Decoding a Video Frame

Create a video session (VkVideoSessionKHR) with codec parameters.
Create video picture resources (reference and output images).
Submit decode operations via vkCmdDecodeVideoKHR.

Example decode command:

VkVideoDecodeInfoKHR decodeInfo = {};
decodeInfo.sType = VK_STRUCTURE_TYPE_VIDEO_DECODE_INFO_KHR;
decodeInfo.srcBuffer = bitstreamBuffer;
decodeInfo.srcBufferOffset = bitstreamOffset;
decodeInfo.srcBufferRange = bitstreamSize;
decodeInfo.dstPictureResource = pictureResource; // reference to image view
decodeInfo.pSetupReferenceSlots = ...; // for DPB

vkCmdDecodeVideoKHR(commandBuffer, &decodeInfo);

You'll need to manage the decoded picture buffer (DPB) and reference frames.

5.4 Encoding

Similarly, you can encode frames by providing raw images and receiving a bitstream.

5.5 Limitations and Considerations

Vulkan Video is still evolving; not all vendors support all codecs.
You need to handle bitstream parsing yourself (the API expects raw NAL units).
It's a low‑level interface; for simpler use, consider libraries like FFmpeg with Vulkan acceleration.

Part 6: Cross‑Platform Considerations

While Metal and Vulkan are powerful, they are platform‑specific. For applications that need to run on both Apple and non‑Apple hardware, you have several options:

6.1 SYCL

SYCL (as covered in previous tutorials) is a high‑level C++ abstraction that can target multiple backends including OpenCL, Level Zero, and CUDA. With SYCL 2020, you can write a single source that compiles to various devices. However, SYCL does not yet directly target Metal or Vulkan compute, but it can target OpenCL which runs on many platforms (including macOS via Intel GPU or with limitations). For Apple Silicon, SYCL is not a direct fit; you'd need to use OpenCL (which is deprecated on macOS) or a SYCL implementation that supports Metal (none currently).

6.2 OpenCL

OpenCL is widely supported (including on Intel GPUs, AMD, NVIDIA), but Apple has deprecated OpenCL on macOS in favor of Metal. For cross‑platform, you might write OpenCL kernels and use them on non‑Apple platforms, and rewrite for Metal on Apple. This is double the work.

6.3 Vulkan Everywhere?

Vulkan runs on Windows, Linux, Android, and macOS via MoltenVK. So you could write your compute kernels in Vulkan and use MoltenVK to run on macOS. MoltenVK translates Vulkan to Metal, and for compute, it works quite well. This gives you a single codebase for all platforms. However, you lose some Metal‑specific optimizations, but the portability is valuable.

For ray tracing, Vulkan ray tracing extensions are not yet supported through MoltenVK (Metal ray tracing is different). So if you need ray tracing on Apple, you'd need separate Metal code.

6.4 Recommendations

If you target only Apple: Use Metal and MPS for maximum performance.
If you target multiple platforms and need compute only: Use Vulkan with MoltenVK for macOS, or SYCL with appropriate backends.
If you need ray tracing across platforms: Write separate backends for Metal and Vulkan, as the APIs differ significantly.
If you need video encode/decode: Vulkan Video is cross‑vendor but not yet on Apple; use platform‑specific APIs (VideoToolbox on macOS, MediaFoundation on Windows, VAAPI on Linux).

Conclusion

You've now explored the advanced corners of Metal and Vulkan: the high‑level performance primitives of MPS, the excitement of ray tracing on both platforms, the efficiency of subgroup operations, and the emerging field of Vulkan Video. Armed with this knowledge, you can build applications that push the boundaries of what's possible on modern GPUs.

Remember, the key to mastery is practice. Implement a small path tracer in Metal, try a subgroup‑accelerated reduction in Vulkan, decode a video stream with Vulkan Video, and see how MPS can accelerate your machine learning models. The possibilities are endless.

If you have questions or want to share your own projects, drop a comment below. Happy coding!

Advanced GPU Optimization: Metal & Vulkan Compute from zero to hero

Javad — Sat, 14 Feb 2026 16:38:13 +0000

Welcome to the exciting world of GPU compute programming! In this comprehensive guide, we'll explore two powerful, low‑level APIs: Metal for Apple devices (especially Apple Silicon) and Vulkan Compute for cross‑platform high‑performance computing. By the end, you'll be able to write efficient compute kernels, manage GPU resources, and optimize for real‑world tasks like matrix multiplication and image processing.

Whether you're a macOS/iOS developer wanting to leverage the Neural Engine and GPU on Apple Silicon, or a cross‑platform developer aiming for maximum performance on Windows, Linux, or Android, this tutorial has you covered.

Introduction: Why Metal and Vulkan Compute?

Metal

Metal is Apple's graphics and compute API, available on all modern Apple devices (iPhone, iPad, Mac). With the advent of Apple Silicon (M1, M2, M3 families), the GPU is tightly integrated with the CPU, offering unified memory and exceptional performance. Metal provides a low‑overhead, explicit API that gives you fine‑grained control over the GPU.

Vulkan Compute

Vulkan is a cross‑platform graphics and compute API from the Khronos Group. Its compute capabilities are exposed through compute pipelines and shaders written in GLSL or HLSL (compiled to SPIR‑V). Vulkan is designed for high‑performance, low‑latency applications and runs on Windows, Linux, Android, macOS (via MoltenVK), and more.

Both Metal and Vulkan follow a similar explicit, command‑buffer‑based model. Understanding one makes learning the other easier. Let's dive in!

Part 1: Metal on Apple Silicon

1.1 Setting Up Metal

To develop with Metal, you'll need:

A Mac with Xcode installed (Xcode includes the Metal compiler, framework, and debugging tools).
An Apple Silicon Mac (M1 or later) for native testing; Intel Macs also work but performance characteristics differ.
For iOS/tvOS, you'll need an Apple Developer account (free for device testing).

Create a new Xcode project (e.g., a command‑line tool for macOS) and add the Metal framework. In your source file, import Metal:

#import <Metal/Metal.h>
// or in Swift: import Metal

1.2 Metal Shading Language (MSL)

Metal uses its own C++‑based shading language, MSL. For compute, you write kernels (functions with kernel keyword). Basic MSL features:

Data types: float, int, half, vectors (float4, int2), matrices (float4x4).
Address spaces: device (global memory), threadgroup (shared memory within a workgroup), constant (read‑only), thread (private).
Built‑in variables:
- uint3 [[thread_position_in_grid]] – global thread ID.
- uint3 [[thread_position_in_threadgroup]] – local thread ID within threadgroup.
- uint3 [[threadgroup_position_in_grid]] – threadgroup ID.
- uint3 [[threads_per_threadgroup]] – size of threadgroup.

Example kernel that adds two vectors:

#include <metal_stdlib>
using namespace metal;

kernel void vecAdd(device const float* a [[buffer(0)]],
                   device const float* b [[buffer(1)]],
                   device float* c [[buffer(2)]],
                   uint id [[thread_position_in_grid]]) {
    c[id] = a[id] + b[id];
}

Note the use of [[buffer(n)]] to bind resources.

1.3 Metal Compute Pipeline

The host side (Objective‑C or Swift) sets up the pipeline:

MTLDevice: The abstract representation of the GPU.
MTLCommandQueue: Queue for submitting command buffers.
MTLComputePipelineState: Compiled compute pipeline (kernel + device).
MTLCommandBuffer: Container for commands.
MTLComputeCommandEncoder: Encodes compute commands.
Dispatch: Specify threadgroup dimensions.

1.4 Your First Metal Compute Program: Vector Addition

Let's walk through the host code in Objective‑C. We'll assume you have a Metal device and have compiled the kernel source.

Step 1: Get the default device

id<MTLDevice> device = MTLCreateSystemDefaultDevice();

Step 2: Create a command queue

id<MTLCommandQueue> commandQueue = [device newCommandQueue];

Step 3: Load the kernel

// Assume you have a .metal file in your project named "kernel.metal"
id<MTLLibrary> library = [device newDefaultLibrary];
id<MTLFunction> kernelFunction = [library newFunctionWithName:@"vecAdd"];

Step 4: Create the pipeline state

NSError *error = nil;
id<MTLComputePipelineState> pipelineState = [device newComputePipelineStateWithFunction:kernelFunction error:&error];

Step 5: Create buffers

NSUInteger dataSize = VEC_SIZE * sizeof(float);

id<MTLBuffer> bufferA = [device newBufferWithBytes:h_a length:dataSize options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferB = [device newBufferWithBytes:h_b length:dataSize options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferC = [device newBufferWithLength:dataSize options:MTLResourceStorageModeShared];

MTLResourceStorageModeShared means the buffer is accessible by both CPU and GPU (unified memory on Apple Silicon).

Step 6: Create command buffer and encoder

id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
id<MTLComputeCommandEncoder> encoder = [commandBuffer computeCommandEncoder];
[encoder setComputePipelineState:pipelineState];
[encoder setBuffer:bufferA offset:0 atIndex:0];
[encoder setBuffer:bufferB offset:0 atIndex:1];
[encoder setBuffer:bufferC offset:0 atIndex:2];

Step 7: Dispatch threads

MTLSize threadgroupSize = MTLSizeMake(256, 1, 1); // threads per threadgroup
MTLSize threadgroupCount = MTLSizeMake((VEC_SIZE + 255) / 256, 1, 1);
[encoder dispatchThreadgroups:threadgroupCount threadsPerThreadgroup:threadgroupSize];

Step 8: End encoding and commit

[encoder endEncoding];
[commandBuffer commit];
[commandBuffer waitUntilCompleted]; // or use completion handler

Step 9: Read results

float *c_ptr = (float*)[bufferC contents];
// c_ptr now contains the result

Step 10: Verify and clean up (ARC will handle objects).

That's it! You've just run your first Metal compute kernel.

1.5 Memory Management in Metal

Metal buffers are created with a storage mode:

MTLResourceStorageModeShared: CPU and GPU access same memory (ideal for Apple Silicon).
MTLResourceStorageModePrivate: Only GPU accessible; used for intermediate buffers.
MTLResourceStorageModeManaged: For discrete GPUs (Intel Macs); requires explicit synchronization.

On Apple Silicon, use Shared for simplicity and performance.

You can also create textures (id<MTLTexture>) for image data. Textures are useful for compute kernels that need spatial locality.

1.6 Performance Considerations

Threadgroup memory (shared memory)

Metal provides threadgroup memory (like CUDA shared memory) for fast data exchange within a workgroup. Declare it in the kernel:

kernel void myKernel(device float* in [[buffer(0)]],
                     device float* out [[buffer(1)]],
                     threadgroup float* shared [[threadgroup(0)]],
                     uint tid [[thread_position_in_threadgroup]],
                     uint gid [[thread_position_in_grid]]) {
    // use shared[tid] ...
}

On the host, you specify the size of threadgroup memory when dispatching.

Coalescing and bank conflicts

Threads in a threadgroup access memory. Ensure that within a warp (32 threads on Apple GPUs), accesses are coalesced. Avoid bank conflicts in threadgroup memory by accessing different banks.

Occupancy and threadgroup size

Use pipelineState.threadExecutionWidth to get the optimal threadgroup size multiple. Typical sizes are multiples of 32.

1.7 Advanced Metal Compute

Using textures

Compute kernels can read/write textures. Useful for image processing.

kernel void sobel(texture2d<float, access::read> inTexture [[texture(0)]],
                  texture2d<float, access::write> outTexture [[texture(1)]],
                  uint2 gid [[thread_position_in_grid]]) {
    float4 color = inTexture.read(gid);
    // compute...
    outTexture.write(result, gid);
}

Multiple dispatches and synchronization

You can encode multiple dispatches in one command buffer, and even use MTLComputeCommandEncoder's setThreadgroupMemoryLength to adjust per‑dispatch.

Profiling

Use Xcode's GPU Debugger and Metal System Trace to analyze performance, view occupancy, and find bottlenecks.

1.8 Example: Tiled Matrix Multiplication in Metal

Let's implement a tiled matrix multiplication kernel using threadgroup memory.

Kernel (MSL):

kernel void matMulTiled(device const float* A [[buffer(0)]],
                        device const float* B [[buffer(1)]],
                        device float* C [[buffer(2)]],
                        constant uint& N [[buffer(3)]],
                        threadgroup float* As [[threadgroup(0)]],
                        threadgroup float* Bs [[threadgroup(1)]],
                        uint2 tgid [[threadgroup_position_in_grid]],
                        uint2 lid [[thread_position_in_threadgroup]],
                        uint2 dims [[threads_per_threadgroup]]) {
    uint tileSize = dims.x; // assuming square threadgroup
    uint row = tgid.y * tileSize + lid.y;
    uint col = tgid.x * tileSize + lid.x;

    float sum = 0.0f;
    for (uint tile = 0; tile < N / tileSize; ++tile) {
        // Load tile of A and B
        As[lid.y * tileSize + lid.x] = A[row * N + (tile * tileSize + lid.x)];
        Bs[lid.y * tileSize + lid.x] = B[(tile * tileSize + lid.y) * N + col];
        threadgroup_barrier(mem_flags::mem_threadgroup);

        // Compute partial product
        for (uint k = 0; k < tileSize; ++k) {
            sum += As[lid.y * tileSize + k] * Bs[k * tileSize + lid.x];
        }
        threadgroup_barrier(mem_flags::mem_threadgroup);
    }
    C[row * N + col] = sum;
}

Host setup:

Buffer A, B, C.
Set N as a constant buffer.
Dispatch with threadgroup size (16,16) and grid size (N/16, N/16).

This kernel leverages threadgroup memory for data reuse, significantly improving performance.

Part 2: Vulkan Compute

2.1 Setting Up Vulkan

To develop Vulkan compute applications, you'll need:

Vulkan SDK from LunarG. Install and set VULKAN_SDK environment variable.
A Vulkan‑capable GPU and drivers (most modern GPUs from NVIDIA, AMD, Intel).
For macOS, you can use MoltenVK (included in the SDK) which translates Vulkan to Metal. Performance is good for compute.

Create a C++ project and include vulkan/vulkan.h. Link against the Vulkan loader (vulkan-1 on Windows, vulkan on Linux).

2.2 Vulkan Compute Pipeline

The Vulkan compute pipeline involves several steps:

Create a VkInstance (with validation layers for debugging).
Pick a physical device (VkPhysicalDevice) that supports compute.
Create a logical device (VkDevice) with a compute queue.
Create a command pool and allocate command buffers.
Create shader module from SPIR‑V.
Create descriptor set layout and pipeline layout.
Create compute pipeline.
Create buffers and allocate device memory.
Update descriptor sets to bind buffers.
Record command buffer: bind pipeline, bind descriptor sets, dispatch.
Submit to queue and synchronize.

Let's go through each step with code.

2.3 GLSL for Compute Shaders

We'll write compute shaders in GLSL (or HLSL compiled to SPIR‑V). A simple vector addition shader:

#version 450
layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;

layout(binding = 0) buffer InputA { float data[]; } a;
layout(binding = 1) buffer InputB { float data[]; } b;
layout(binding = 2) buffer OutputC { float data[]; } c;

void main() {
    uint id = gl_GlobalInvocationID.x;
    c.data[id] = a.data[id] + b.data[id];
}

Compile this to SPIR‑V using glslangValidator or glslc. For example:

glslc -c -o vecAdd.spv vecAdd.comp

2.4 Memory and Resources

Vulkan uses VkBuffer objects backed by VkDeviceMemory. You must allocate memory, bind it, and map it for host access.

Descriptor sets are used to bind resources (buffers, images) to shaders. You need:

VkDescriptorSetLayout describing the bindings.
VkDescriptorPool to allocate descriptor sets.
VkDescriptorSet containing actual buffer references.

Push constants are small amounts of data that can be set directly in command buffers (fast).

2.5 Your First Vulkan Compute Program: Vector Addition

Let's outline the host code in C++. Error checking omitted for brevity.

Step 1: Create instance and pick device (standard Vulkan initialization). We'll assume you have a VkDevice and a compute queue.

Step 2: Create shader module

VkShaderModuleCreateInfo createInfo = {};
createInfo.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO;
createInfo.codeSize = codeSize;
createInfo.pCode = (uint32_t*)spvCode;
vkCreateShaderModule(device, &createInfo, nullptr, &shaderModule);

Step 3: Create descriptor set layout (bindings for three buffers)

VkDescriptorSetLayoutBinding bindings[3] = {};
bindings[0].binding = 0; bindings[0].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER; bindings[0].descriptorCount = 1; bindings[0].stageFlags = VK_SHADER_STAGE_COMPUTE_BIT;
// similarly for 1 and 2
VkDescriptorSetLayoutCreateInfo layoutInfo = {};
layoutInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
layoutInfo.bindingCount = 3; layoutInfo.pBindings = bindings;
vkCreateDescriptorSetLayout(device, &layoutInfo, nullptr, &descriptorSetLayout);

Step 4: Create pipeline layout

VkPipelineLayoutCreateInfo pipelineLayoutInfo = {};
pipelineLayoutInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO;
pipelineLayoutInfo.setLayoutCount = 1; pipelineLayoutInfo.pSetLayouts = &descriptorSetLayout;
vkCreatePipelineLayout(device, &pipelineLayoutInfo, nullptr, &pipelineLayout);

Step 5: Create compute pipeline

VkComputePipelineCreateInfo pipelineInfo = {};
pipelineInfo.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO;
pipelineInfo.stage.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
pipelineInfo.stage.stage = VK_SHADER_STAGE_COMPUTE_BIT;
pipelineInfo.stage.module = shaderModule;
pipelineInfo.stage.pName = "main";
pipelineInfo.layout = pipelineLayout;
vkCreateComputePipelines(device, VK_NULL_HANDLE, 1, &pipelineInfo, nullptr, &pipeline);

Step 6: Create buffers and allocate memory

VkBufferCreateInfo bufferInfo = {};
bufferInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
bufferInfo.size = dataSize;
bufferInfo.usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; // for copying from host
bufferInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
vkCreateBuffer(device, &bufferInfo, nullptr, &bufferA);
// Similarly for B, C

// Get memory requirements, allocate memory, bind buffers
VkMemoryRequirements memReqs;
vkGetBufferMemoryRequirements(device, bufferA, &memReqs);
VkMemoryAllocateInfo allocInfo = {};
allocInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
allocInfo.allocationSize = memReqs.size;
allocInfo.memoryTypeIndex = findMemoryType(memReqs.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT);
vkAllocateMemory(device, &allocInfo, nullptr, &memoryA);
vkBindBufferMemory(device, bufferA, memoryA, 0);

Step 7: Map memory and copy initial data

void* mapped;
vkMapMemory(device, memoryA, 0, dataSize, 0, &mapped);
memcpy(mapped, h_a, dataSize);
vkUnmapMemory(device, memoryA);
// Similarly for B

Step 8: Create descriptor pool and descriptor set

VkDescriptorPoolSize poolSize = {};
poolSize.type = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
poolSize.descriptorCount = 3;
VkDescriptorPoolCreateInfo poolInfo = {};
poolInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
poolInfo.maxSets = 1;
poolInfo.poolSizeCount = 1; poolInfo.pPoolSizes = &poolSize;
vkCreateDescriptorPool(device, &poolInfo, nullptr, &descriptorPool);

VkDescriptorSetAllocateInfo allocInfo = {};
allocInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO;
allocInfo.descriptorPool = descriptorPool;
allocInfo.descriptorSetCount = 1; allocInfo.pSetLayouts = &descriptorSetLayout;
vkAllocateDescriptorSets(device, &allocInfo, &descriptorSet);

Step 9: Update descriptor set with buffer info

VkDescriptorBufferInfo bufferInfoA = { bufferA, 0, dataSize };
VkWriteDescriptorSet writeA = {};
writeA.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
writeA.dstSet = descriptorSet;
writeA.dstBinding = 0;
writeA.descriptorCount = 1;
writeA.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
writeA.pBufferInfo = &bufferInfoA;
// Similarly for B, C (bindings 1,2)
vkUpdateDescriptorSets(device, 3, writes, 0, nullptr);

Step 10: Create command pool and command buffer

VkCommandPoolCreateInfo poolCmdInfo = {};
poolCmdInfo.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO;
poolCmdInfo.queueFamilyIndex = computeQueueFamilyIndex;
vkCreateCommandPool(device, &poolCmdInfo, nullptr, &commandPool);

VkCommandBufferAllocateInfo cmdAllocInfo = {};
cmdAllocInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO;
cmdAllocInfo.commandPool = commandPool;
cmdAllocInfo.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY;
cmdAllocInfo.commandBufferCount = 1;
vkAllocateCommandBuffers(device, &cmdAllocInfo, &commandBuffer);

Step 11: Record commands

VkCommandBufferBeginInfo beginInfo = {};
beginInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
vkBeginCommandBuffer(commandBuffer, &beginInfo);

vkCmdBindPipeline(commandBuffer, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline);
vkCmdBindDescriptorSets(commandBuffer, VK_PIPELINE_BIND_POINT_COMPUTE, pipelineLayout, 0, 1, &descriptorSet, 0, nullptr);

uint32_t groupCountX = (VEC_SIZE + 255) / 256;
vkCmdDispatch(commandBuffer, groupCountX, 1, 1);

vkEndCommandBuffer(commandBuffer);

Step 12: Submit and wait

VkSubmitInfo submitInfo = {};
submitInfo.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO;
submitInfo.commandBufferCount = 1; submitInfo.pCommandBuffers = &commandBuffer;
vkQueueSubmit(computeQueue, 1, &submitInfo, VK_NULL_HANDLE);
vkQueueWaitIdle(computeQueue);

Step 13: Read back results (map memory of buffer C and copy).

That's the complete flow. It's verbose, but gives you full control.

2.6 Synchronization

Vulkan uses fences (GPU‑CPU sync) and semaphores (GPU‑GPU sync). In compute, you often just wait for the queue idle. For finer control, use vkWaitForFences after submitting with a fence.

Memory dependencies are handled by pipeline barriers. For example, if you write to a buffer then read it in a later dispatch, you need a barrier to ensure visibility.

2.7 Performance Optimization

Workgroup size

Choose a workgroup size that is a multiple of the subgroup size (typically 32 or 64) and that maximizes occupancy. Use VkPhysicalDeviceProperties::maxComputeWorkGroupSize.

Shared memory

Declare shared memory in GLSL with shared float tile[16][16]; and use barrier() for synchronization. Similar to Metal threadgroup memory.

Subgroup operations

Vulkan exposes subgroup operations via the VK_EXT_subgroup extension. These allow efficient reductions, scans, and shuffles within a subgroup (like warp intrinsics).

Avoiding bank conflicts – same principle as Metal.

2.8 Example: Tiled Matrix Multiplication in Vulkan

Shader (GLSL):

#version 450
layout(local_size_x = 16, local_size_y = 16) in;

layout(binding = 0) buffer InputA { float data[]; } A;
layout(binding = 1) buffer InputB { float data[]; } B;
layout(binding = 2) buffer OutputC { float data[]; } C;
layout(push_constant) uniform PushConsts { uint N; } pc;

shared float As[16][16];
shared float Bs[16][16];

void main() {
    uint row = gl_GlobalInvocationID.y;
    uint col = gl_GlobalInvocationID.x;
    uint localRow = gl_LocalInvocationID.y;
    uint localCol = gl_LocalInvocationID.x;

    float sum = 0.0;
    for (uint tile = 0; tile < pc.N / 16; ++tile) {
        As[localRow][localCol] = A.data[row * pc.N + (tile * 16 + localCol)];
        Bs[localRow][localCol] = B.data[(tile * 16 + localRow) * pc.N + col];
        barrier();

        for (uint k = 0; k < 16; ++k) {
            sum += As[localRow][k] * Bs[k][localCol];
        }
        barrier();
    }
    C.data[row * pc.N + col] = sum;
}

Host changes:

Use push constant to pass N.
Dispatch with grid size (N/16, N/16) and local size (16,16).
Ensure descriptor set binds buffers A, B, C.

Part 3: Comparative Analysis and Cross‑Platform Strategies

3.1 Metal vs Vulkan Compute: Similarities and Differences

Aspect	Metal	Vulkan Compute
API Style	Objective‑C / Swift (objects, references)	C (handles, structures)
Shader Language	MSL (C++‑based)	GLSL / HLSL compiled to SPIR‑V
Resource Binding	[[buffer(n)]] and [[texture(n)]]	Descriptor sets with bindings
Memory Model	Unified on Apple Silicon; discrete otherwise	Explicit memory management
Workgroup/Threadgroup	`threadgroup` memory, `threadgroup_barrier`	`shared` memory, `barrier()`
Error Handling	NSError, exceptions (Swift)	Return codes (VkResult)
Debugging/Profiling	Xcode GPU Debugger, Metal System Trace	RenderDoc, Vulkan Validation Layers, GPU vendors' tools

Both are explicit, low‑level, and provide fine‑grained control. Metal is slightly simpler to use, especially on Apple platforms, while Vulkan offers cross‑platform portability.

3.2 Writing Portable Compute Code

If you need to support both Metal and Vulkan, consider:

Writing compute kernels in a common language (e.g., using a library like OpenCL C or SYCL that compiles to both, but that's beyond this guide).
Maintaining two separate codebases with shared algorithms.
Using conditional compilation and abstraction layers (like MoltenVK to run Vulkan on Metal, though compute performance is good).

For compute kernels, the core logic is often identical; only the syntax for resource binding and built‑in variables differs. You can use macros to abstract differences.

Example kernel snippet (pseudo‑code):

#ifdef METAL
#define GLOBAL_ID [[thread_position_in_grid]]
#define SHARED threadgroup
#define BARRIER threadgroup_barrier(mem_flags::mem_threadgroup)
#else
#define GLOBAL_ID gl_GlobalInvocationID
#define SHARED shared
#define BARRIER barrier()
#endif

Part 4: Advanced Topics

4.1 Indirect Dispatch

Both APIs support indirect dispatch where the workgroup counts are read from a buffer, allowing GPU‑generated workloads.

Metal: [encoder dispatchThreadgroupsWithIndirectBuffer:indirectBuffer indirectBufferOffset:0 threadsPerThreadgroup:threadgroupSize];

Vulkan: vkCmdDispatchIndirect with a buffer containing the dispatch dimensions.

4.2 GPU Timestamps and Queries

Metal: Use MTLCounterSampleBuffer (on supported devices) to sample GPU timestamps.

Vulkan: Use query pools (VK_QUERY_TYPE_TIMESTAMP) to record timestamps in command buffers.

4.3 Interop with Graphics Pipelines

Both APIs allow mixing compute and graphics in the same command buffer. For example, you can use a compute shader to generate data for a vertex buffer, then render it.

4.4 Multi‑GPU (Vulkan) and Multi‑Device (Metal)

Vulkan supports multi‑GPU via logical devices with multiple physical devices, but it's complex. Metal has MTLDevice per GPU; you can create separate queues and manage workloads manually.

4.5 Real‑world Example: Sobel Edge Detection

Let's outline a Sobel kernel in both APIs.

Metal Sobel (texture version):

kernel void sobel(texture2d<float, access::read> inTex [[texture(0)]],
                  texture2d<float, access::write> outTex [[texture(1)]],
                  uint2 gid [[thread_position_in_grid]]) {
    if (gid.x < outTex.get_width() && gid.y < outTex.get_height()) {
        float sumX = 0, sumY = 0;
        for (int dy = -1; dy <= 1; dy++) {
            for (int dx = -1; dx <= 1; dx++) {
                uint2 coord = uint2(gid.x + dx, gid.y + dy);
                float val = inTex.read(coord).r;
                sumX += val * sobelX[dy+1][dx+1];
                sumY += val * sobelY[dy+1][dx+1];
            }
        }
        float mag = min(1.0f, hypot(sumX, sumY));
        outTex.write(float4(mag, mag, mag, 1), gid);
    }
}

Vulkan Sobel (buffer version, assuming image stored as float array):

#version 450
layout(local_size_x = 16, local_size_y = 16) in;
layout(binding = 0) buffer Input { float pixels[]; } inBuf;
layout(binding = 1) buffer Output { float pixels[]; } outBuf;
layout(push_constant) uniform Dims { uint width; uint height; } dims;

void main() {
    uint x = gl_GlobalInvocationID.x;
    uint y = gl_GlobalInvocationID.y;
    if (x >= dims.width || y >= dims.height) return;
    float sumX = 0, sumY = 0;
    for (int dy = -1; dy <= 1; dy++) {
        for (int dx = -1; dx <= 1; dx++) {
            int nx = int(x) + dx, ny = int(y) + dy;
            if (nx < 0 || nx >= dims.width || ny < 0 || ny >= dims.height) continue;
            float val = inBuf.pixels[ny * dims.width + nx];
            sumX += val * sobelX[dy+1][dx+1];
            sumY += val * sobelY[dy+1][dx+1];
        }
    }
    float mag = min(1.0, length(vec2(sumX, sumY)));
    outBuf.pixels[y * dims.width + x] = mag;
}

Conclusion

Congratulations! You've journeyed through the depths of Metal compute on Apple Silicon and Vulkan compute. You now know how to set up the environment, write kernels, manage memory, optimize performance, and implement complex patterns like matrix multiplication and edge detection.

Next Steps

Metal: Explore Metal Performance Shaders (MPS) for highly optimized primitives (convolution, matrix multiplication, etc.). Look into Metal Ray Tracing for advanced graphics.
Vulkan: Dive into the Vulkan specification, experiment with subgroup operations, and try the Vulkan Ray Tracing extensions. Explore Vulkan Video for encode/decode.
Cross‑platform: Consider using SYCL or OpenCL for broader portability, but now you have the low‑level knowledge to understand what's happening under the hood.

The world of GPU compute is vast and exciting. Keep experimenting, profiling, and optimizing. And remember – the best way to learn is to write code. So fire up Xcode or your Vulkan SDK and start computing!

If you found this guide helpful, share it with your fellow developers. Have questions or want to share your own experiences? Leave a comment below. Happy coding!

Advanced GPU Optimization: OpenCL & SYCL from zero to hero - Part 2

Javad — Sat, 14 Feb 2026 16:31:54 +0000

Welcome back, heterogeneous computing enthusiasts! In our previous guides, we covered the fundamentals of OpenCL and SYCL, from setting up your environment to writing basic kernels and optimizing them with local memory. Now it's time to go deeper. In this advanced tutorial, we'll explore:

Key features of the OpenCL specification (2.x/3.0) that enable sophisticated programming.
What SYCL 2020 brings to the table, including unified shared memory and group algorithms.
Implementation of complex parallel patterns: reductions, scans, and stencils.
How to target CPUs, GPUs, and FPGAs with SYCL and compare performance.
A tour of oneAPI's domain‑specific libraries (oneMKL, oneDNN, oneDPL, etc.) that supercharge your development.

Whether you're looking to squeeze every drop of performance from your hardware or write portable code that scales across architectures, this guide has you covered.

Part 1: Deep Dive into the OpenCL Specification

The OpenCL specification has evolved significantly. While OpenCL 1.2 is the most widely supported baseline, modern devices support newer features that can greatly simplify and accelerate development. Let's look at the highlights.

1.1 OpenCL 2.x Features

Shared Virtual Memory (SVM)

OpenCL 2.0 introduced SVM, allowing the host and devices to share pointers. This eliminates explicit buffer mappings and can simplify programming. Three types exist:

Coarse-grained SVM: Sharing at buffer granularity; synchronization needed.
Fine-grained SVM: Sharing at any memory location; optional atomics.
Fine-grained buffer SVM: Like fine-grained but limited to a buffer region.

With SVM, you can pass complex data structures (like linked lists) to kernels without deep copies.

Device-side Enqueue

Kernels can enqueue other kernels, creating nested parallelism. This is great for recursive algorithms or dynamic workloads. The parent kernel creates a child kernel and enqueues it to a device queue.

Pipes

Pipes are FIFO data structures connecting kernels. They facilitate streaming patterns without global memory round trips. One kernel writes to a pipe, another reads from it, with synchronization handled by the hardware.

Work-group Collective Functions

OpenCL 2.0 added built‑in work‑group functions like work_group_reduce_add, work_group_scan_exclusive_add, etc., enabling efficient intra‑group reductions and scans.

C11 Atomics

Kernels can use atomic operations conforming to the C11 memory model, essential for lock‑free algorithms.

1.2 OpenCL 3.0: The New Baseline

OpenCL 3.0, released in 2020, takes a different approach: it makes OpenCL 1.2 the mandatory baseline, with all 2.x features optional. This allows vendors to implement only what they support while maintaining compatibility. Key additions:

A new query mechanism to check for optional features.
Support for newer C++ features in kernels (like auto, lambda expressions) via the cl_khr_c++11 extension.
Better integration with Vulkan and other APIs.

1.3 Using Advanced Features in Practice

To use these features, you must check for extensions and enable them at program creation. For example, to use SVM:

cl_device_svm_capabilities caps;
clGetDeviceInfo(device, CL_DEVICE_SVM_CAPABILITIES, sizeof(caps), &caps, NULL);
if (caps & CL_DEVICE_SVM_COARSE_GRAIN_BUFFER) {
    // Allocate coarse‑grain SVM buffer
    clSVMAlloc(context, CL_MEM_READ_WRITE, size, 0);
}

Device‑side enqueue requires building the program with the -cl-std=CL2.0 flag and using the cl_khr_device_enqueue extension.

While these features are powerful, they are not universally supported across all devices. Always fall back to portable code when necessary.

Part 2: Deep Dive into the SYCL 2020 Specification

SYCL 2020 is a major update that aligns with modern C++ and adds many features inspired by OpenCL 2.x and beyond. Let's explore the most impactful additions.

2.1 Unified Shared Memory (USM)

SYCL 2020 standardizes USM, providing pointer‑based memory management. Three allocation types:

malloc_device: Device‑accessible memory (not directly on host).
malloc_host: Host‑accessible memory (may be cached on device).
malloc_shared: Automatically migrated memory, accessible from both sides.

Example:

queue q;
int *data = malloc_shared<int>(N, q);
q.parallel_for(range<1>(N), [=](id<1> i) { data[i] = i; }).wait();
// data is updated on host after synchronization
free(data, q);

USM simplifies porting CUDA/HIP code and reduces verbosity compared to buffers.

2.2 Reductions

SYCL 2020 introduces built‑in reduction support. You can use reduction objects with parallel_for to perform reductions efficiently.

queue q;
std::vector<float> vec(N, 1.0f);
buffer<float> buf(vec);
float init = 0.0f;
{
    buffer<float> sum_buf(&init, 1);
    q.submit([&](handler& h) {
        auto in = buf.get_access<access::mode::read>(h);
        auto out = sum_buf.get_access<access::mode::write>(h);
        h.parallel_for(range<1>(N), reduction(out, 0.0f, std::plus<>()), [=](id<1> i, auto& sum) {
            sum += in[i];
        });
    });
}
// sum_buf now contains the total

The implementation chooses the most efficient strategy (tree reduction, atomics, etc.) based on the device.

2.3 Group Algorithms and Sub‑Groups

SYCL 2020 provides collective algorithms for work‑groups, similar to OpenCL 2.0:

group_barrier()
group_broadcast()
group_reduce(), group_scan() (exclusive/inclusive)
sub_group class for warp/wavefront operations.

Example of group reduction:

h.parallel_for(nd_range<1>(N, 256), [=](nd_item<1> it) {
    auto g = it.get_group();
    int local_sum = ...;
    int group_sum = reduce_over_group(g, local_sum, std::plus<>());
    if (it.get_local_id(0) == 0)
        atomic_ref<int, memory_order::relaxed, memory_scope::device>(global_sum) += group_sum;
});

Sub‑groups allow portable SIMD programming. You can query sub‑group size and use shuffles.

2.4 In‑Order Queues and Properties

SYCL 2020 adds queue properties to specify in‑order execution, which can simplify dependencies:

queue q{ property::queue::in_order() };

2.5 Interoperability with OpenCL

SYCL 2020 improves interop: you can create a SYCL platform, device, context, or queue from native OpenCL handles, and vice versa. This allows incremental migration or mixing low‑level control with high‑level productivity.

Part 3: Complex Parallel Patterns

Now that we've covered the specifications, let's implement three fundamental parallel patterns: reduction, scan, and stencil. We'll show both OpenCL and SYCL versions, with optimizations.

3.1 Reduction (Sum of an Array)

Reduction combines elements using an associative operator (e.g., sum). A common approach is hierarchical: each work‑group reduces its chunk in local memory, then one thread per group adds to global atomic.

OpenCL kernel:

__kernel void reduce(__global const float *in, __global float *out, __local float *localMem, int n) {
    int gid = get_global_id(0);
    int lid = get_local_id(0);
    int groupSize = get_local_size(0);
    int groupId = get_group_id(0);

    // Load data into local memory
    float sum = (gid < n) ? in[gid] : 0.0f;
    localMem[lid] = sum;
    barrier(CLK_LOCAL_MEM_FENCE);

    // Reduce within group
    for (int stride = groupSize / 2; stride > 0; stride >>= 1) {
        if (lid < stride) {
            localMem[lid] += localMem[lid + stride];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Write partial sum
    if (lid == 0) {
        out[groupId] = localMem[0];
    }
}

After this kernel, you have partial sums. Run a second kernel with one work‑group to combine them, or copy to host and finalize.

SYCL with reductions:
SYCL's built‑in reduction handles this automatically, but for educational purposes, here's a manual version:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 1, access::mode::read_write, access::target::local> localMem(range<1>(256), h);

    h.parallel_for(nd_range<1>(n, 256), [=](nd_item<1> it) {
        int gid = it.get_global_id(0);
        int lid = it.get_local_id(0);
        float sum = (gid < n) ? in[gid] : 0.0f;
        localMem[lid] = sum;
        it.barrier();

        for (int stride = 128; stride > 0; stride >>= 1) {
            if (lid < stride) {
                localMem[lid] += localMem[lid + stride];
            }
            it.barrier();
        }

        if (lid == 0) {
            out[it.get_group(0)] = localMem[0];
        }
    });
});

But the SYCL 2020 way is simpler:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    h.parallel_for(range<1>(n), reduction(sum_buf, h, 0.0f, std::plus<>()), [=](id<1> i, auto& sum) {
        sum += in[i];
    });
});

3.2 Scan (Prefix Sum)

Scan computes cumulative sums. An efficient work‑group scan uses a double‑buffered local memory approach.

OpenCL kernel for inclusive scan (one block):

__kernel void scan(__global const float *in, __global float *out, __local float *temp, int n) {
    int gid = get_global_id(0);
    int lid = get_local_id(0);
    int groupSize = get_local_size(0);
    int groupId = get_group_id(0);

    // Load data
    temp[2 * lid] = (gid < n) ? in[gid] : 0;
    temp[2 * lid + 1] = (gid + groupSize < n) ? in[gid + groupSize] : 0;
    barrier(CLK_LOCAL_MEM_FENCE);

    // Upsweep phase (parallel reduction)
    for (int stride = 1; stride < 2 * groupSize; stride <<= 1) {
        int index = (lid + 1) * stride * 2 - 1;
        if (index < 2 * groupSize && lid < groupSize) {
            temp[index] += temp[index - stride];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Downsweep phase
    for (int stride = groupSize; stride > 0; stride >>= 1) {
        int index = (lid + 1) * stride * 2 - 1;
        if (index + stride < 2 * groupSize && lid < groupSize) {
            temp[index + stride] += temp[index];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Write results
    if (gid < n) {
        out[gid] = temp[2 * lid];
        if (gid + groupSize < n)
            out[gid + groupSize] = temp[2 * lid + 1];
    }
}

After this, you need a second pass to add the total of previous groups. For large arrays, you'd combine with a top‑level scan of group sums.

SYCL using group_scan:
SYCL 2020 provides group_scan_inclusive and group_scan_exclusive:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 1, access::mode::read_write, access::target::local> localMem(range<1>(256), h);

    h.parallel_for(nd_range<1>(n, 256), [=](nd_item<1> it) {
        int gid = it.get_global_id(0);
        int lid = it.get_local_id(0);
        float val = (gid < n) ? in[gid] : 0.0f;
        auto grp = it.get_group();
        float scan_val = group_scan_inclusive(grp, val, std::plus<>());
        if (gid < n)
            out[gid] = scan_val;
    });
});

This performs an inclusive scan within each work‑group. To get a global scan, you'd again need to handle the group offsets.

3.3 Stencil (2D Heat Conduction)

Stencils update each element based on its neighbors. Optimizations: use local memory to cache a tile plus halo.

OpenCL kernel for 2D heat stencil:

__kernel void heat(__global const float *in, __global float *out, int width, int height) {
    int x = get_global_id(0);
    int y = get_global_id(1);
    int localX = get_local_id(0);
    int localY = get_local_id(1);
    int localSizeX = get_local_size(0);
    int localSizeY = get_local_size(1);

    // Local memory with halo of 1
    __local float tile[LOCAL_SIZE_Y + 2][LOCAL_SIZE_X + 2];

    // Load interior
    if (x < width && y < height)
        tile[localY + 1][localX + 1] = in[y * width + x];
    else
        tile[localY + 1][localX + 1] = 0.0f;

    // Load halo regions (simplified: assumes boundary conditions)
    if (localY == 0 && y > 0)
        tile[0][localX + 1] = in[(y-1) * width + x];
    if (localY == localSizeY - 1 && y < height - 1)
        tile[localSizeY + 1][localX + 1] = in[(y+1) * width + x];
    // similarly for left/right
    barrier(CLK_LOCAL_MEM_FENCE);

    if (x < width && y < height) {
        float center = tile[localY + 1][localX + 1];
        float left   = tile[localY + 1][localX];
        float right  = tile[localY + 1][localX + 2];
        float up     = tile[localY][localX + 1];
        float down   = tile[localY + 2][localX + 1];
        out[y * width + x] = center + 0.125f * (left + right + up + down - 4.0f * center);
    }
}

SYCL version (with local accessor and nd_range):

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 2, access::mode::read_write, access::target::local> tile(range<2>(LOCAL_Y+2, LOCAL_X+2), h);

    h.parallel_for(nd_range<2>(range<2>(height, width), range<2>(LOCAL_Y, LOCAL_X)), [=](nd_item<2> it) {
        int x = it.get_global_id(1);
        int y = it.get_global_id(0);
        int lx = it.get_local_id(1);
        int ly = it.get_local_id(0);

        // Load interior
        if (x < width && y < height)
            tile[ly+1][lx+1] = in[y][x];

        // Load halos (simplified)
        // ... similar to OpenCL
        it.barrier();

        if (x < width && y < height) {
            float center = tile[ly+1][lx+1];
            float left   = tile[ly+1][lx];
            float right  = tile[ly+1][lx+2];
            float up     = tile[ly][lx+1];
            float down   = tile[ly+2][lx+1];
            out[y][x] = center + 0.125f * (left + right + up + down - 4.0f * center);
        }
    });
});

These patterns are building blocks for many scientific and engineering applications.

Part 4: SYCL on Different Hardware and Performance Comparison

One of SYCL's greatest promises is "write once, run anywhere." But how does performance compare across CPUs, GPUs, and FPGAs? Let's explore the practical considerations.

4.1 Targeting Devices with SYCL

Selecting a device is straightforward:

// Default selector (picks a device, often GPU if available)
queue q;

// Specific selectors
cpu_selector cpu;
gpu_selector gpu;
accelerator_selector fpga; // for FPGA

queue q_cpu(cpu);
queue q_gpu(gpu);
queue q_fpga(fpga);

You can also write custom selectors based on device info (e.g., preferring a GPU with larger local memory).

4.2 Performance Considerations per Device

CPU:

Optimize for vectorization: use wide SIMD (work‑group size = vector width).
Minimize data transfers; CPU shares memory with host, so USM can be efficient.
Avoid excessive local memory; CPU caches are good.

GPU:

Maximize occupancy: choose work‑group size that hides latency (e.g., 128‑256).
Use local memory for data reuse.
Ensure coalesced global memory accesses.

FPGA:

Kernels are synthesized into hardware pipelines.
Loop unrolling, pipelining, and on‑chip memory are key.
SYCL for FPGA supports [[intel::max_work_group_size(1)]] to create single‑work‑item kernels that are highly pipelined.
Use #pragma unroll and [[intel::initiation_interval(1)]] directives.

4.3 Benchmarking and Comparison

To compare performance across devices, you need a common benchmark. Let's take matrix multiplication (tiled) and run it on different hardware.

Methodology:

Compile with appropriate flags: for CPU, use -O3 -march=native; for GPU, -O3 -fsycl; for FPGA, you need the Intel FPGA SDK and use -Xshardware for actual hardware runs.
Measure kernel execution time using SYCL events:

q.submit([&](handler& h) {
    // ...
}).wait();
auto start = std::chrono::steady_clock::now();
q.submit([&](handler& h) {
    // kernel
}).wait();
auto end = std::chrono::steady_clock::now();

Better: use SYCL profiling info via property::queue::enable_profiling().

Expected Observations:

CPU: Good for small matrices, latency may be higher due to OS scheduling.
GPU: Excels at large, regular computations.
FPGA: Often slower for compute‑bound tasks but can be very power‑efficient and excel at pipeline‑friendly streaming.

Case Study: Matrix Multiplication
We can run the tiled SYCL kernel from Part 2.7 on a CPU, GPU, and FPGA (if available). The performance will vary dramatically. For example, on an Intel Core i9, the CPU might achieve ~50 GFLOPs; on an Intel UHD Graphics, maybe ~200 GFLOPs; on an Arria 10 FPGA, perhaps ~50 GFLOPs but with lower power. However, the FPGA could achieve lower latency for small batches and better predictability.

4.4 Portable Optimization Tips

Use #ifdef SYCL_DEVICE_CPU etc. to specialize for each architecture.
Tune work‑group size dynamically using device.get_info<info::device::max_work_group_size>().
For FPGA, design kernels to be highly parallel with pipeline parallelism rather than data parallelism.

Part 5: oneAPI Domain-Specific Libraries

oneAPI includes a rich set of libraries that provide optimized building blocks for common tasks. Using them can drastically accelerate development while ensuring high performance across devices.

5.1 oneMKL (Math Kernel Library)

oneMKL provides BLAS, LAPACK, FFT, RNG, and more. It has SYCL interfaces that accept SYCL buffers or USM pointers.

Example: Matrix multiplication using oneMKL:

#include <oneapi/mkl/blas.hpp>
queue q;
float alpha = 1.0f, beta = 0.0f;
oneapi::mkl::blas::gemm(q, oneapi::mkl::transpose::nontrans, oneapi::mkl::transpose::nontrans,
                        N, N, N, alpha, d_A, N, d_B, N, beta, d_C, N);
q.wait();

This will call an optimized GEMM for the target device (CPU, GPU, or FPGA). No need to write your own kernel!

FFT example:

#include <oneapi/mkl/dft.hpp>
using namespace oneapi::mkl::dft;
descriptor<precision::single, domain::real> transform(N);
transform.commit(q);
// in and out buffers...
transform.compute_forward(in, out);

5.2 oneDNN (Deep Neural Networks)

oneDNN (formerly DNNL) provides primitives for deep learning: convolution, pooling, normalization, etc. It has SYCL and OpenCL backends.

Example: Convolution:

#include <oneapi/dnnl/dnnl.hpp>
using namespace dnnl;

engine eng(engine::kind::gpu, 0);
stream s(eng);
memory::dims conv_src_tz = {N, C, H, W};
memory::dims conv_weights_tz = {K, C, R, S};
memory::dims conv_dst_tz = {N, K, H', W'};
// Create memory objects
auto conv_src_mem = memory({{conv_src_tz}, memory::data_type::f32, memory::format_tag::any}, eng);
// Initialize convolution descriptor and primitive
convolution_forward::desc conv_desc(prop_kind::forward, algorithm::convolution_direct,
                                    conv_src_md, conv_weights_md, conv_dst_md,
                                    strides, padding_l, padding_r);
convolution_forward::primitive_desc conv_pd(conv_desc, eng);
auto conv = convolution_forward(conv_pd);
conv.execute(s, {{DNNL_ARG_SRC, conv_src_mem}, {DNNL_ARG_WEIGHTS, conv_weights_mem}, {DNNL_ARG_DST, conv_dst_mem}});
s.wait();

oneDNN automatically selects the best implementation for the hardware.

5.3 oneDPL (DPC++ Library)

oneDPL provides parallel algorithms similar to C++17's Parallel STL, implemented with SYCL.

Example: parallel sort:

#include <oneapi/dpl/algorithm>
#include <oneapi/dpl/execution>

std::vector<float> data(N);
// fill data...
auto policy = oneapi::dpl::execution::make_device_policy(q);
std::sort(policy, data.begin(), data.end());

Other algorithms: reduce, transform, scan, etc.

5.4 oneTBB (Threading Building Blocks)

While not exclusively for devices, oneTBB integrates with oneAPI to provide task parallelism on the host, which can coordinate with device work.

5.5 Other oneAPI Libraries

oneVPL – Video Processing Library for encode/decode.
oneDAL – Data Analytics Library.
oneCCL – Collective Communications Library for distributed computing.

5.6 Combining Libraries in a Project

A typical SYCL + oneAPI project might use:

oneDPL for data preparation on CPU.
oneMKL for FFT on GPU.
oneDNN for inference on FPGA.
oneTBB to manage host-side tasks.

All with the same SYCL queue and memory model.

Conclusion

We've journeyed through the advanced features of OpenCL and SYCL, implemented essential parallel patterns, explored performance across diverse hardware, and surveyed the powerful oneAPI libraries. Armed with this knowledge, you can now tackle complex computational problems with confidence and write code that runs efficiently on CPUs, GPUs, and FPGAs alike.

Remember, the key to mastering heterogeneous computing is practice. Experiment with these patterns on your own hardware, profile them, and see where optimizations take you. And when you need a productivity boost, don't reinvent the wheel – leverage the oneAPI libraries.

Further Resources:

Thank you for reading! If you have questions or want to share your experiences with OpenCL, SYCL, or oneAPI, leave a comment below. Happy coding!

Advanced GPU Optimization: OpenCL & SYCL from zero to hero

Javad — Sat, 14 Feb 2026 16:29:50 +0000

Welcome to the world of heterogeneous programming! In this comprehensive guide, we’ll explore two powerful frameworks that let you harness the computational power of CPUs, GPUs, FPGAs, and other accelerators: OpenCL and SYCL (the heart of Intel’s oneAPI initiative). Whether you’re a student, researcher, or industry developer, by the end of this tutorial you’ll be able to write portable, efficient code that runs on a variety of hardware.

We’ll start from the absolute basics (no prior experience required) and progress to advanced topics like local memory optimization, profiling, and multi-device programming. Along the way, you’ll see how OpenCL provides a low‑level, C‑based interface for fine‑grained control, while SYCL offers a modern C++ single‑source approach that builds on OpenCL’s concepts.

Let’s dive in!

Introduction: Why OpenCL and SYCL?

OpenCL (Open Computing Language) is the industry standard for heterogeneous computing. It allows you to write kernels (programs) that execute on any OpenCL‑compliant device—CPUs, GPUs, DSPs, FPGAs—using a C‑based language. It gives you explicit control over memory and execution, but at the cost of some verbosity.

SYCL (pronounced “sickle”) is a higher‑level C++ abstraction built on top of OpenCL. It enables single‑source programming: you write host and device code in the same C++ file, using modern C++ features. SYCL is the foundation of Intel’s oneAPI initiative, which aims to provide a unified programming model across CPUs, GPUs, and FPGAs. SYCL code can be compiled for different backends (OpenCL, Level Zero, CUDA) without modification.

Both are essential tools in the heterogeneous computing landscape. Learning them gives you the flexibility to target a wide range of hardware while keeping your code portable.

Part 1: OpenCL Fundamentals

1.1 Setting Up OpenCL

Before we write code, we need an OpenCL development environment. OpenCL consists of a host API (to manage devices, memory, and execution) and a kernel language (based on C99 with extensions).

Installation:

On Linux: Install the OpenCL headers and a runtime (e.g., from your GPU vendor). For Intel CPUs/GPUs, install the Intel Compute Runtime. For NVIDIA, install the CUDA toolkit which includes OpenCL. For AMD, install ROCm or the AMD APP SDK.
On Windows: Download and install the appropriate SDK from your hardware vendor (Intel, NVIDIA, AMD). You can also use the open‑source PoCL (Portable Computing Language) for CPU development.

Basic includes and linking:

#include <CL/cl.h>

Link against the OpenCL library (e.g., -lOpenCL on Linux, OpenCL.lib on Windows).

1.2 The OpenCL Execution Model

OpenCL organizes work into an NDRange (N‑dimensional range). The key concepts:

Work‑item: The smallest unit of execution. Each work‑item executes the same kernel but works on different data.
Work‑group: A collection of work‑items that execute together on a compute unit. Work‑items in the same group can synchronize via barriers and share memory through local memory.
NDRange: The entire grid of work‑items, partitioned into work‑groups. You specify the global size (total work‑items) and local size (work‑group size).

When you launch a kernel, you provide the number of dimensions (1, 2, or 3), the global size in each dimension, and optionally the local size. The runtime then schedules work‑groups onto compute units.

1.3 OpenCL Memory Model

OpenCL defines several memory regions:

Global memory: Accessible by all work‑items; large but high latency.
Constant memory: Read‑only global memory, cached.
Local memory: Shared within a work‑group; fast, programmer‑managed.
Private memory: Per work‑item, typically registers.

Host code manages device memory via buffer objects. Data is explicitly moved between host and device.

1.4 Your First OpenCL Program: Vector Addition

Let’s write a simple vector addition kernel and the host code to run it.

Kernel (stored in a string or separate .cl file):

__kernel void vecAdd(__global const float *a,
                     __global const float *b,
                     __global float *c) {
    int i = get_global_id(0);
    c[i] = a[i] + b[i];
}

Host code (C):

#include <stdio.h>
#include <stdlib.h>
#include <CL/cl.h>

#define VEC_SIZE 1048576  // 1M elements

int main() {
    // Step 1: Get platform and device
    cl_platform_id platform;
    cl_device_id device;
    clGetPlatformIDs(1, &platform, NULL);
    clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &device, NULL);

    // Step 2: Create context and command queue
    cl_context context = clCreateContext(NULL, 1, &device, NULL, NULL, NULL);
    cl_command_queue queue = clCreateCommandQueue(context, device, 0, NULL);

    // Step 3: Allocate host memory and initialize data
    size_t bytes = VEC_SIZE * sizeof(float);
    float *h_a = (float*)malloc(bytes);
    float *h_b = (float*)malloc(bytes);
    float *h_c = (float*)malloc(bytes);
    for (int i = 0; i < VEC_SIZE; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    // Step 4: Create device buffers
    cl_mem d_a = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, bytes, h_a, NULL);
    cl_mem d_b = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, bytes, h_b, NULL);
    cl_mem d_c = clCreateBuffer(context, CL_MEM_WRITE_ONLY, bytes, NULL, NULL);

    // Step 5: Build the program
    const char *kernelSource = "__kernel void vecAdd(__global const float *a, __global const float *b, __global float *c) { int i = get_global_id(0); c[i] = a[i] + b[i]; }";
    cl_program program = clCreateProgramWithSource(context, 1, &kernelSource, NULL, NULL);
    clBuildProgram(program, 1, &device, NULL, NULL, NULL);
    cl_kernel kernel = clCreateKernel(program, "vecAdd", NULL);

    // Step 6: Set kernel arguments
    clSetKernelArg(kernel, 0, sizeof(cl_mem), &d_a);
    clSetKernelArg(kernel, 1, sizeof(cl_mem), &d_b);
    clSetKernelArg(kernel, 2, sizeof(cl_mem), &d_c);

    // Step 7: Execute kernel
    size_t global_size = VEC_SIZE;
    size_t local_size = 256;  // must divide global_size
    clEnqueueNDRangeKernel(queue, kernel, 1, NULL, &global_size, &local_size, 0, NULL, NULL);

    // Step 8: Read result back
    clEnqueueReadBuffer(queue, d_c, CL_TRUE, 0, bytes, h_c, 0, NULL, NULL);

    // Step 9: Verify (simplified)
    for (int i = 0; i < 10; i++)
        printf("%f + %f = %f\n", h_a[i], h_b[i], h_c[i]);

    // Step 10: Clean up
    clReleaseMemObject(d_a); clReleaseMemObject(d_b); clReleaseMemObject(d_c);
    clReleaseKernel(kernel); clReleaseProgram(program);
    clReleaseCommandQueue(queue); clReleaseContext(context);
    free(h_a); free(h_b); free(h_c);

    return 0;
}

Compile and run (on Linux):

gcc -o vecAdd vecAdd.c -lOpenCL
./vecAdd

Explanation:

We query a platform and device (here, we take the first GPU).
Create a context and command queue (the queue is where we enqueue commands).
Allocate host buffers and device buffers. clCreateBuffer with CL_MEM_COPY_HOST_PTR copies data immediately.
Build the program from source string; in real applications, you’d load from a file and check build errors.
Set kernel arguments, enqueue the kernel with global and local sizes.
Read the result back to host with clEnqueueReadBuffer.
Clean up OpenCL objects.

This is the basic pattern for any OpenCL program.

1.5 Building and Running OpenCL Programs

Always check error codes. Each OpenCL function returns a cl_int error code. Use macros like:

#define CL_CHECK(err) \
    if (err != CL_SUCCESS) { \
        fprintf(stderr, "OpenCL error %d at %s:%d\n", err, __FILE__, __LINE__); \
        exit(EXIT_FAILURE); \
    }

// Usage:
cl_int err;
cl_context context = clCreateContext(NULL, 1, &device, NULL, NULL, &err);
CL_CHECK(err);

When building a program, you should check build logs:

clBuildProgram(program, 1, &device, NULL, NULL, NULL);
if (err != CL_SUCCESS) {
    size_t log_size;
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, 0, NULL, &log_size);
    char *log = (char*)malloc(log_size);
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, log_size, log, NULL);
    fprintf(stderr, "Build log:\n%s\n", log);
    free(log);
    exit(EXIT_FAILURE);
}

1.6 Profiling OpenCL with Events

OpenCL events can measure execution time. Enable profiling on the command queue:

cl_command_queue queue = clCreateCommandQueue(context, device, CL_QUEUE_PROFILING_ENABLE, &err);

Then attach an event to kernel launch and read timestamps:

cl_event event;
clEnqueueNDRangeKernel(queue, kernel, 1, NULL, &global_size, &local_size, 0, NULL, &event);
clWaitForEvents(1, &event);

cl_ulong start, end;
clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_START, sizeof(cl_ulong), &start, NULL);
clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_END, sizeof(cl_ulong), &end, NULL);
double time_ms = (end - start) / 1.0e6;
printf("Kernel execution time: %f ms\n", time_ms);

clReleaseEvent(event);

1.7 Example: Matrix Multiplication (Naive and Tiled)

Let's implement matrix multiplication to demonstrate local memory usage.

Naive kernel (each work-item computes one element of C):

__kernel void matMulNaive(__global const float *A,
                          __global const float *B,
                          __global float *C,
                          int N) {
    int row = get_global_id(1);
    int col = get_global_id(0);
    if (row < N && col < N) {
        float sum = 0.0f;
        for (int k = 0; k < N; k++) {
            sum += A[row * N + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

Host side (simplified):

size_t global_size[2] = {N, N};
clEnqueueNDRangeKernel(queue, kernel, 2, NULL, global_size, NULL, 0, NULL, NULL);

Tiled kernel using local memory:

#define TILE_SIZE 16

__kernel void matMulTiled(__global const float *A,
                          __global const float *B,
                          __global float *C,
                          int N) {
    __local float As[TILE_SIZE][TILE_SIZE];
    __local float Bs[TILE_SIZE][TILE_SIZE];

    int row = get_global_id(1);
    int col = get_global_id(0);
    int localRow = get_local_id(1);
    int localCol = get_local_id(0);

    float sum = 0.0f;
    for (int tile = 0; tile < N / TILE_SIZE; tile++) {
        // Load tile of A and B into local memory
        As[localRow][localCol] = A[row * N + (tile * TILE_SIZE + localCol)];
        Bs[localRow][localCol] = B[(tile * TILE_SIZE + localRow) * N + col];
        barrier(CLK_LOCAL_MEM_FENCE);

        // Compute partial sum
        for (int k = 0; k < TILE_SIZE; k++) {
            sum += As[localRow][k] * Bs[k][localCol];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

Launch configuration:

size_t local_size[2] = {TILE_SIZE, TILE_SIZE};
size_t global_size[2] = {N, N};
clEnqueueNDRangeKernel(queue, kernel, 2, NULL, global_size, local_size, 0, NULL, NULL);

This tiled version reduces global memory traffic by reusing data loaded into local memory. It's a key optimization pattern.

1.8 Advanced OpenCL Topics

Multiple devices: You can create a context with multiple devices and use separate command queues.
Image objects: OpenCL supports 2D/3D images with samplers for graphics‑oriented workloads.
SVM (Shared Virtual Memory): Allows sharing pointers between host and device (OpenCL 2.0).
Device-side enqueue: Kernels can enqueue other kernels (OpenCL 2.0).

Part 2: SYCL (oneAPI) Fundamentals

2.1 What is SYCL?

SYCL is a C++ abstraction layer that sits on top of OpenCL (and other backends). It allows you to write host and device code in the same file using standard C++17 (or later). Key features:

Single-source: Kernel code is written as C++ functors or lambdas, compiled by a SYCL compiler.
High-level abstractions: Buffers and accessors manage memory dependencies; command groups encapsulate kernel launches.
Portability: Same code can target CPUs, GPUs, FPGAs, etc., via different SYCL implementations (Intel oneAPI DPC++, ComputeCpp, hipSYCL).
Interoperability: Can mix with OpenCL code if needed.

Intel’s oneAPI is a complete suite of tools and libraries built around SYCL (DPC++ – Data Parallel C++). We'll focus on DPC++ as the most widely adopted SYCL implementation.

2.2 Setting Up oneAPI / SYCL

Option 1: Intel oneAPI base toolkit (free):

Download from Intel oneAPI.
Install and source the environment: source /opt/intel/oneapi/setvars.sh.

Option 2: Open-source DPC++:

Follow instructions on intel/llvm to build the DPC++ compiler.

Option 3: hipSYCL (supports NVIDIA and AMD backends):

See hipSYCL documentation.

We'll assume you have dpcpp (DPC++ compiler) available.

2.3 SYCL Execution Model

SYCL programs revolve around:

queue: Submits work to a device (similar to OpenCL command queue).
buffer: Manages data across host and device.
accessor: Requests access to a buffer inside a command group; specifies access mode (read, write, read_write) and target (host, device).
handler: Used inside command groups to set kernel arguments and launch parallel work.
nd_range: Defines global and local size (like OpenCL NDRange).
parallel_for: Launches a kernel over a range.

A typical SYCL program:

#include <CL/sycl.hpp>
using namespace sycl;

int main() {
    queue q;  // selects default device

    buffer<float, 1> a_buf{ h_a, range<1>{N} };
    buffer<float, 1> b_buf{ h_b, range<1>{N} };
    buffer<float, 1> c_buf{ range<1>{N} };

    q.submit([&](handler& h) {
        auto a = a_buf.get_access<access::mode::read>(h);
        auto b = b_buf.get_access<access::mode::read>(h);
        auto c = c_buf.get_access<access::mode::write>(h);

        h.parallel_for(range<1>{N}, [=](id<1> i) {
            c[i] = a[i] + b[i];
        });
    });

    // Accessor on host to read result
    auto c_host = c_buf.get_access<access::mode::read>();
    // ... use c_host ...
}

2.4 Memory Model: Buffers and Accessors

Buffers abstract data movement. Accessors create dependencies: when you request an accessor in a command group, the runtime ensures data is available on the device before the kernel runs, and updates the host after.

USM (Unified Shared Memory) is an alternative that uses pointers, similar to CUDA/HIP. SYCL supports device USM, shared USM, and host USM.

2.5 Your First SYCL Program: Vector Addition

Let's rewrite the vector addition using SYCL.

#include <CL/sycl.hpp>
#include <iostream>
#include <vector>

using namespace sycl;

int main() {
    const int N = 1 << 20;
    std::vector<float> a(N), b(N), c(N);
    for (int i = 0; i < N; i++) {
        a[i] = i * 1.0f;
        b[i] = (i + 1) * 1.0f;
    }

    try {
        queue q;  // default device

        // Create buffers from host data (moves data to device automatically)
        buffer<float, 1> a_buf(a.data(), range<1>(N));
        buffer<float, 1> b_buf(b.data(), range<1>(N));
        buffer<float, 1> c_buf(c.data(), range<1>(N));

        // Submit command group
        q.submit([&](handler& h) {
            // Request accessors
            auto a_acc = a_buf.get_access<access::mode::read>(h);
            auto b_acc = b_buf.get_access<access::mode::read>(h);
            auto c_acc = c_buf.get_access<access::mode::write>(h);

            // Launch kernel
            h.parallel_for(range<1>(N), [=](id<1> i) {
                c_acc[i] = a_acc[i] + b_acc[i];
            });
        });

        // Buffers go out of scope, so data is copied back to host automatically
        // (destructor of c_buf waits for completion and updates host)
    } catch (sycl::exception& e) {
        std::cerr << "SYCL exception: " << e.what() << std::endl;
        return 1;
    }

    // Verify
    for (int i = 0; i < 10; i++)
        std::cout << a[i] << " + " << b[i] << " = " << c[i] << std::endl;

    return 0;
}

Compile with:

dpcpp -o vecAdd vecAdd.cpp
./vecAdd

Explanation:

queue q creates a queue to the default device. You can select a specific device using device_selector.
Buffers wrap host data; they don't copy immediately but manage data lazily.
Inside the command group, accessors specify how the kernel uses the data. The runtime ensures data is on the device before execution.
The kernel is a lambda capturing accessors by value. It executes in parallel over a 1D range.
When buffers are destroyed (at end of scope), they wait for any dependent operations and copy data back to host if needed.

2.6 Error Handling and Asynchronous Exceptions

SYCL uses exceptions for synchronous errors. Asynchronous errors (during kernel execution) can be caught via an async handler:

void async_handler(sycl::exception_list e) {
    for (auto& ep : e) {
        try { std::rethrow_exception(ep); }
        catch (sycl::exception& e) { std::cerr << "Async SYCL exception: " << e.what() << std::endl; }
    }
}

queue q(device_selector{}, async_handler);

2.7 Example: Matrix Multiplication with SYCL

Now let's implement matrix multiplication using SYCL's nd_range for work‑groups and local memory.

#include <CL/sycl.hpp>
#include <iostream>
#include <vector>

using namespace sycl;

#define TILE_SIZE 16

int main() {
    const int N = 1024;
    std::vector<float> A(N * N), B(N * N), C(N * N, 0.0f);
    // Initialize A, B...

    try {
        queue q;

        buffer<float, 2> a_buf(A.data(), range<2>(N, N));
        buffer<float, 2> b_buf(B.data(), range<2>(N, N));
        buffer<float, 2> c_buf(C.data(), range<2>(N, N));

        q.submit([&](handler& h) {
            auto a = a_buf.get_access<access::mode::read>(h);
            auto b = b_buf.get_access<access::mode::read>(h);
            auto c = c_buf.get_access<access::mode::write>(h);

            // Local memory accessor
            accessor<float, 2, access::mode::read_write, access::target::local> As(TILE_SIZE, TILE_SIZE, h);
            accessor<float, 2, access::mode::read_write, access::target::local> Bs(TILE_SIZE, TILE_SIZE, h);

            h.parallel_for(nd_range<2>({N, N}, {TILE_SIZE, TILE_SIZE}), [=](nd_item<2> it) {
                int row = it.get_global_id(1);
                int col = it.get_global_id(0);
                int localRow = it.get_local_id(1);
                int localCol = it.get_local_id(0);

                float sum = 0.0f;
                for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
                    // Load tiles
                    As[localRow][localCol] = a[row][tile * TILE_SIZE + localCol];
                    Bs[localRow][localCol] = b[tile * TILE_SIZE + localRow][col];
                    it.barrier(access::fence_space::local_space);

                    // Compute
                    for (int k = 0; k < TILE_SIZE; ++k) {
                        sum += As[localRow][k] * Bs[k][localCol];
                    }
                    it.barrier(access::fence_space::local_space);
                }

                if (row < N && col < N) {
                    c[row][col] = sum;
                }
            });
        });
    } catch (sycl::exception& e) {
        std::cerr << e.what() << std::endl;
        return 1;
    }

    // Use C...
    return 0;
}

Notes:

We use 2D buffers with range<2>.
Local memory is allocated via accessor with target::local.
nd_range combines global and local sizes.
Barriers are called on the nd_item to synchronize work‑group.

2.8 Unified Shared Memory (USM)

USM simplifies memory management by using pointers. There are three types:

device: Allocated on device, accessible only from device.
host: Allocated on host, accessible only from host.
shared: Managed pointer that can be accessed from both host and device (migrates data automatically).

Example with shared USM:

queue q;
float *a = malloc_shared<float>(N, q);
float *b = malloc_shared<float>(N, q);
float *c = malloc_shared<float>(N, q);

// Initialize on host
for (int i = 0; i < N; i++) {
    a[i] = i * 1.0f;
    b[i] = (i + 1) * 1.0f;
}

q.parallel_for(range<1>(N), [=](id<1> i) {
    c[i] = a[i] + b[i];
}).wait();

// c is ready on host
// ...

free(a, q); free(b, q); free(c, q);

USM is more convenient for porting CUDA/HIP code but requires explicit waits or dependencies.

2.9 Advanced SYCL Topics

Device selection: Use default_selector, gpu_selector, cpu_selector, or custom.
Kernel programming model: SYCL supports basic parallel_for, nd_range, and single_task.
Reductions: SYCL 2020 adds built‑in reductions.
Graph dependencies: Use depends_on and events.
Interoperability with OpenCL: SYCL can create buffers from cl_mem, etc.

Part 3: Advanced Topics and Optimization

3.1 Work-Group Size and Occupancy

Both OpenCL and SYCL rely on work‑groups (or blocks) to map to hardware compute units. Choosing the right local size is crucial.

OpenCL: Local size must divide global size evenly (or use clEnqueueNDRangeKernel with NULL local size to let runtime choose). Typically, use multiples of the preferred work‑group size from device info.
SYCL: Local size specified in nd_range. You can query device info: device.get_info<info::device::max_work_group_size>().

A good rule of thumb: start with a work‑group size of 64–256 and tune based on profiling.

3.2 Local Memory and Barriers

OpenCL: Use __local pointers or __local arrays. Synchronize with barrier(CLK_LOCAL_MEM_FENCE).
SYCL: Use local accessors. Synchronize with item.barrier(access::fence_space::local_space).

Local memory is key for data reuse (e.g., tiled matrix multiplication) and reduction.

3.3 Memory Access Patterns

Coalescing: Work‑items in a work‑group should access consecutive global memory addresses to maximize bandwidth. In OpenCL, this means the global id (or get_global_id(0)) should be the fastest‑changing dimension.
Avoiding bank conflicts: In local memory, ensure that within a work‑group, different work‑items access different banks (or all access the same address for broadcast). This may require padding.

3.4 Using SIMD and Vectorization

OpenCL kernels can use vector types (float4, int4, etc.) to explicitly use SIMD. The compiler may also auto‑vectorize. In SYCL, you can use the vec class.

3.5 Profiling and Debugging

OpenCL:

Use events as shown earlier.
Vendor tools: Intel VTune, AMD CodeXL, NVIDIA Visual Profiler (with OpenCL support).

SYCL:

oneAPI provides Intel VTune Profiler and Intel Advisor.
The queue can be constructed with property::queue::enable_profiling() to get kernel times via events.

Debugging:

OpenCL: printf inside kernels works (if supported). Use clBuildProgram logs.
SYCL: Use std::cout in kernels (SYCL 2020 supports device iostream emulation, but it's slow). For serious debugging, use Intel's Debugger for OpenCL or gdb‑oneapi.

3.6 Interoperability: OpenCL and SYCL

SYCL can interoperate with OpenCL objects:

Create SYCL platform/device from cl_platform_id/cl_device_id.
Create SYCL queue from cl_command_queue.
Create SYCL buffer from cl_mem.

This allows incremental porting or mixing low‑level control with high‑level productivity.

3.7 Targeting Different Devices

OpenCL: Code can run on any OpenCL device, but performance tuning may be device‑specific.
SYCL with oneAPI: You can target CPU, GPU, FPGA with the same source. For FPGA, SYCL supports pipeline‑oriented optimizations via [[intel::kernel_args_restrict]] and other attributes.

Example: selecting a GPU device explicitly:

std::vector<device> devices = device::get_devices(info::device_type::gpu);
queue q(devices[0]);

Part 4: Portability and Choosing Between OpenCL and SYCL

4.1 OpenCL Strengths and Weaknesses

Strengths:

Mature, widely supported.
Fine‑grained control over devices and memory.
C‑based kernel language (easy to learn for C programmers).

Weaknesses:

Verbose host code.
No single‑source, so host and device code are separate.
Kernel language lacks modern C++ features.

4.2 SYCL Strengths

Strengths:

Single‑source C++ with lambdas and templates.
RAII and automatic dependency management via buffers.
Portability across backends (OpenCL, Level Zero, CUDA).
Backed by oneAPI ecosystem with optimized libraries.

Weaknesses:

Newer, still evolving (SYCL 2020 is a major step).
Compiler support not as universal as OpenCL.
Learning curve for the abstraction model.

4.3 When to Use Which

Use OpenCL if you need to support very old hardware, have strict control requirements, or are maintaining an existing OpenCL codebase.
Use SYCL/oneAPI for new projects where productivity and portability are priorities, especially if you target Intel GPUs/FPGAs or want to keep options open for AMD/NVIDIA via hipSYCL.

Conclusion

You've now journeyed through the fundamentals of both OpenCL and SYCL, from setting up your environment to writing optimized kernels for vector addition and matrix multiplication. You've seen how OpenCL gives you explicit control, while SYCL provides a modern C++ interface that simplifies many aspects of heterogeneous programming.

Both frameworks are powerful tools in the developer's toolbox. As you continue, explore real‑world applications, contribute to open‑source projects, and leverage the rich ecosystem of libraries (oneMKL, oneDNN, etc.). The world of heterogeneous computing is vast, and you're now equipped to navigate it.

Next Steps:

Dive into the OpenCL specification and SYCL 2020 specification.
Experiment with more complex patterns: reductions, scans, stencils.
Try SYCL on different hardware (CPU, GPU, FPGA) and compare performance.
Explore oneAPI's domain‑specific libraries for deep learning, ray tracing, etc.

If you found this guide valuable, share it with your peers. Have questions or want to share your experiences? Leave a comment below. Happy coding!

Advanced GPU Optimization: CUDA & HIP from zero to hero

Javad — Sat, 14 Feb 2026 16:23:19 +0000

Welcome, fellow developer! If you’ve ever wanted to tap into the immense power of graphics processors for general-purpose computing, you’re in the right place. GPU programming used to be an NVIDIA-only game with CUDA, but the rise of AMD GPUs in workstations, servers, and supercomputers has made vendor lock‑in a thing of the past. Enter HIP – a C++ runtime API that lets you write code that runs on both NVIDIA and AMD GPUs with minimal changes.

In this comprehensive guide, we’ll start from the very basics (no prior GPU experience assumed) and work our way up to advanced topics like streams, shared memory optimizations, and multi‑GPU programming. Along the way, you’ll learn CUDA inside out and then discover how HIP gives you portability without sacrificing performance. By the end, you’ll be comfortable writing efficient, portable GPU kernels and ready to tackle real‑world parallel problems.

Let’s dive in!

Part 1: CUDA Fundamentals

1.1 Setting Up CUDA

Before we write any code, we need a working CUDA development environment.

Check your hardware: You need an NVIDIA GPU with Compute Capability 3.0 or higher. Run lspci | grep -i nvidia on Linux or check Device Manager on Windows.
Install the CUDA Toolkit:
- Download the appropriate installer from NVIDIA CUDA Toolkit.
- Follow the installation guide for your OS.
Verify installation:

   nvcc --version

You should see the CUDA version. Also run nvidia-smi to check driver and GPU status.

Set environment variables (if not done automatically):

   export PATH=/usr/local/cuda/bin:$PATH
   export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Now you’re ready to compile CUDA code with nvcc.

1.2 The CUDA Programming Model

A GPU contains hundreds or thousands of cores. To use them effectively, we organize work into a hierarchy:

Thread: The smallest unit of execution. Each thread runs the same kernel function but works on different data.
Block: A group of threads that can cooperate via shared memory and synchronize with __syncthreads(). Threads in the same block run on the same streaming multiprocessor (SM).
Grid: A collection of blocks that together cover the entire problem.

When you launch a kernel, you specify the grid and block dimensions (in 1D, 2D, or 3D). Each thread gets unique indices:

threadIdx.x, threadIdx.y, threadIdx.z – position inside its block.
blockIdx.x, blockIdx.y, blockIdx.z – position of the block in the grid.
blockDim.x, blockDim.y, blockDim.z – size of each block.
gridDim.x, gridDim.y, gridDim.z – size of the grid.

The typical formula to map a thread to a data element is:

int i = blockIdx.x * blockDim.x + threadIdx.x;

1.3 Memory Hierarchy

Global memory: Large (several GB), high latency, accessible by all threads. It persists across kernel launches.
Shared memory: Small (tens of KB per block), low latency, shared among threads in the same block. Programmer‑managed cache.
Registers: Private to each thread, fastest.
Constant memory: Read‑only, cached, good for broadcast data.
Texture memory: Specialized for 2D spatial locality.

Understanding this hierarchy is key to writing fast kernels.

1.4 Your First CUDA Kernel: Vector Addition

Let’s implement a simple vector addition. This will introduce all the essential steps.

#include <iostream>
#include <cuda_runtime.h>

// Kernel definition
__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {  // Guard against extra threads
        c[i] = a[i] + b[i];
    }
}

int main() {
    int n = 1 << 20;  // 1,048,576 elements
    size_t bytes = n * sizeof(float);

    // Allocate host memory
    float *h_a = new float[n];
    float *h_b = new float[n];
    float *h_c = new float[n];

    // Initialize input arrays
    for (int i = 0; i < n; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    // Allocate device memory
    float *d_a, *d_b, *d_c;
    cudaMalloc(&d_a, bytes);
    cudaMalloc(&d_b, bytes);
    cudaMalloc(&d_c, bytes);

    // Copy data from host to device
    cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);
    cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice);

    // Launch kernel
    int threadsPerBlock = 256;
    int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;
    vecAdd<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, n);

    // Copy result back to host
    cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost);

    // Verify (simple check)
    for (int i = 0; i < n; i++) {
        if (h_c[i] != h_a[i] + h_b[i]) {
            std::cout << "Mismatch at index " << i << "\n";
            break;
        }
    }

    // Cleanup
    cudaFree(d_a);
    cudaFree(d_b);
    cudaFree(d_c);
    delete[] h_a;
    delete[] h_b;
    delete[] h_c;

    return 0;
}

Explanation:

__global__ marks a function as a kernel – callable from host and executed on device.
We compute a global index i and, if within bounds, perform the addition.
<<<blocksPerGrid, threadsPerBlock>>> sets the execution configuration.
cudaMalloc allocates memory on the GPU.
cudaMemcpy transfers data between host and device (the last argument specifies direction).
Always free device memory with cudaFree to avoid leaks.

Compile with:

nvcc -o vecAdd vecAdd.cu
./vecAdd

If everything works, you should see no output (or a success message if you add one).

1.5 Understanding Kernel Launch Parameters

The launch configuration <<<grid, block, sharedMem, stream>>> has two required parameters and two optional:

grid (dim3): number of blocks per grid. Can be 1D, 2D, or 3D. E.g., dim3 grid(16, 16) creates a 16x16 grid (256 blocks).
block (dim3): number of threads per block. Usually 1D, but can be 2D/3D. Total threads per block ≤ 1024 (for modern GPUs).
sharedMem (size_t): dynamic shared memory per block (default 0).
stream (cudaStream_t): specifies which stream to enqueue the kernel in (default 0, the null stream).

In our example we used 1D grids and blocks. For 2D data (like images), you might use 2D blocks:

dim3 block(16, 16);
dim3 grid((width + block.x - 1) / block.x, (height + block.y - 1) / block.y);
kernel<<<grid, block>>>(...);

Inside the kernel, you would then compute:

int x = blockIdx.x * blockDim.x + threadIdx.x;
int y = blockIdx.y * blockDim.y + threadIdx.y;
int idx = y * width + x;

1.6 Memory Management in Depth

cudaMalloc(void **ptr, size_t size) – allocates linear memory on device.
cudaFree(void *ptr) – frees device memory.
cudaMemcpy(void *dst, const void *src, size_t count, cudaMemcpyKind kind) – synchronously copies memory.
cudaMemcpyAsync – asynchronous version, requires pinned host memory.
cudaMallocHost – allocates page‑locked (pinned) host memory, which enables faster transfers and asynchronous copies.
cudaMemset – sets device memory to a value.

Pinned memory example:

float *h_a;
cudaMallocHost(&h_a, bytes);   // pinned
cudaMemcpyAsync(d_a, h_a, bytes, cudaMemcpyHostToDevice, stream);

Use pinned memory when you have many transfers or overlapping with computation.

1.7 Error Handling

CUDA functions return an error code of type cudaError_t. You should always check them:

cudaError_t err = cudaMalloc(&d_a, bytes);
if (err != cudaSuccess) {
    std::cerr << "cudaMalloc failed: " << cudaGetErrorString(err) << std::endl;
    exit(EXIT_FAILURE);
}

For kernel launches, which are asynchronous, you can check for launch errors with cudaGetLastError() and wait for completion with cudaDeviceSynchronize() to catch execution errors:

kernel<<<grid, block>>>(args);
err = cudaGetLastError();   // catch launch errors
if (err != cudaSuccess) { ... }
cudaDeviceSynchronize();    // wait and catch execution errors
err = cudaGetLastError();
if (err != cudaSuccess) { ... }

Better yet, wrap calls in a macro:

#define CUDA_CHECK(call) \
    do { \
        cudaError_t err = call; \
        if (err != cudaSuccess) { \
            std::cerr << "CUDA error in " << __FILE__ << ":" << __LINE__ << ": " \
                      << cudaGetErrorString(err) << std::endl; \
            exit(EXIT_FAILURE); \
        } \
    } while(0)

// Usage:
CUDA_CHECK(cudaMalloc(&d_a, bytes));

1.8 Profiling and Debugging Basics

Profiling: NVIDIA provides nvprof (deprecated but still useful) and the newer Nsight Systems / Nsight Compute. To profile a simple app:

  nvprof ./vecAdd

This shows kernel execution time, memory transfer times, etc. For deeper analysis, use Nsight Compute: ncu ./vecAdd.

Debugging: Use cuda-gdb (Linux) or the Visual Studio debugger on Windows. Compile with -g -G to generate debug info:

  nvcc -g -G -o vecAdd vecAdd.cu
  cuda-gdb ./vecAdd

Inside cuda-gdb, you can set breakpoints, inspect threads, and step through device code.

Memory error checking: compute-sanitizer (formerly cuda-memcheck) helps detect out‑of‑bounds accesses, race conditions, etc.

  compute-sanitizer ./vecAdd

1.9 Practice: Matrix Multiplication

Let's solidify our understanding with a classic: matrix multiplication C = A * B (N x N). We'll start with a naive kernel.

Naive implementation (each thread computes one element of C):

__global__ void matMulNaive(const float *A, const float *B, float *C, int N) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    if (row < N && col < N) {
        float sum = 0.0f;
        for (int k = 0; k < N; k++) {
            sum += A[row * N + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

Launch configuration (assuming 16x16 blocks):

dim3 block(16, 16);
dim3 grid((N + block.x - 1) / block.x, (N + block.y - 1) / block.y);
matMulNaive<<<grid, block>>>(d_A, d_B, d_C, N);

Performance issues: This naive version has poor memory access patterns (especially for B, where each thread accesses a different column, causing non‑coalesced reads) and no data reuse. We'll improve it later with shared memory.

Part 2: Intermediate CUDA

Now that you're comfortable with basic CUDA, let's explore techniques to write faster kernels.

2.1 Performance Considerations

Occupancy: The ratio of active warps to maximum possible warps on an SM. Higher occupancy can hide latency but isn't always the goal. Use the CUDA Occupancy Calculator or the cudaOccupancyMaxPotentialBlockSize function to tune block size.

Warp divergence: Threads in a warp execute in lockstep. If they take different paths (e.g., an if‑else), the warp serially executes both paths, reducing performance. Try to keep control flow uniform within warps.

Coalesced memory access: Global memory bandwidth is best utilized when consecutive threads access consecutive memory locations. In our vector addition, thread i accesses a[i] – perfectly coalesced. In naive matrix multiplication, row‑major access of A is coalesced, but column‑major access of B is not. We'll fix that with shared memory.

2.2 Shared Memory and Bank Conflicts

Shared memory is divided into banks. If multiple threads access the same bank (with some exceptions), we get a bank conflict and the accesses are serialized. To avoid conflicts, ensure that threads within a warp access different banks (or all access the same word in broadcast mode). This is an advanced topic, but be aware of it when designing shared memory layouts.

2.3 Tiled Matrix Multiplication with Shared Memory

Let's rewrite matrix multiplication to use shared memory. The idea: load tiles of A and B into shared memory, then compute partial sums.

#define TILE_SIZE 16

__global__ void matMulShared(const float *A, const float *B, float *C, int N) {
    __shared__ float As[TILE_SIZE][TILE_SIZE];
    __shared__ float Bs[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;

    // Identify the row and column of the C element to compute
    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;

    // Loop over all tiles
    for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
        // Load one tile of A and B into shared memory
        As[ty][tx] = A[row * N + (tile * TILE_SIZE + tx)];
        Bs[ty][tx] = B[(tile * TILE_SIZE + ty) * N + col];
        __syncthreads();  // Ensure tile is loaded before computation

        // Compute partial product for this tile
        for (int k = 0; k < TILE_SIZE; ++k) {
            sum += As[ty][k] * Bs[k][tx];
        }
        __syncthreads();  // Avoid overwriting shared memory before next tile
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

What changed?

Each block loads a tile of A and a tile of B into shared memory.
All threads in the block cooperate to load the tiles (coalesced because threads in a warp load consecutive elements from global memory).
Computation uses the fast shared memory.
__syncthreads() ensures that all threads have finished loading before any thread starts using the data, and that no thread starts the next tile before all have finished the current tile.

This kernel can achieve near‑peak performance for large matrices.

2.4 Atomic Operations

When multiple threads need to update the same location (e.g., histogram, reduction), you can use atomic operations to avoid race conditions.

__global__ void histogram(const unsigned char *data, unsigned int *bins, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        atomicAdd(&bins[data[i]], 1);
    }
}

CUDA provides atomicAdd, atomicSub, atomicExch, atomicMin, atomicMax, atomicAnd, atomicOr, atomicXor for various data types.

Performance: Atomics can be slow if many threads contend for the same location. Use shared memory to reduce contention when possible (e.g., each block computes its own partial histogram, then combine with atomics).

2.5 Streams and Concurrency

Streams allow you to overlap kernel execution with data transfers and other operations. A stream is a sequence of operations that execute in order. Operations from different streams may overlap.

Example: overlapping copy and compute:

cudaStream_t stream1, stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

// Split data into two halves
int half = n / 2;

// Asynchronous memory copies (requires pinned host memory)
cudaMemcpyAsync(d_a1, h_a1, half_bytes, cudaMemcpyHostToDevice, stream1);
cudaMemcpyAsync(d_a2, h_a2, half_bytes, cudaMemcpyHostToDevice, stream2);

// Launch kernels in respective streams
kernel<<<grid, block, 0, stream1>>>(d_a1, d_b1, d_c1, half);
kernel<<<grid, block, 0, stream2>>>(d_a2, d_b2, d_c2, half);

// Copy back
cudaMemcpyAsync(h_c1, d_c1, half_bytes, cudaMemcpyDeviceToHost, stream1);
cudaMemcpyAsync(h_c2, d_c2, half_bytes, cudaMemcpyDeviceToHost, stream2);

cudaStreamSynchronize(stream1);
cudaStreamSynchronize(stream2);

cudaStreamDestroy(stream1);
cudaStreamDestroy(stream2);

Note: Use pinned memory (cudaMallocHost) for asynchronous transfers. Also, kernels from different streams may run concurrently if resources permit.

2.6 Events for Timing

CUDA events provide a way to measure elapsed time on the device accurately.

cudaEvent_t start, stop;
cudaEventCreate(&start);
cudaEventCreate(&stop);

cudaEventRecord(start);
kernel<<<grid, block>>>(args);
cudaEventRecord(stop);

cudaEventSynchronize(stop);  // Wait for the stop event to be recorded
float milliseconds = 0;
cudaEventElapsedTime(&milliseconds, start, stop);
std::cout << "Kernel time: " << milliseconds << " ms\n";

cudaEventDestroy(start);
cudaEventDestroy(stop);

2.7 Unified Memory

Unified Memory (UM) creates a single pointer accessible from both CPU and GPU. Data is automatically migrated on demand. It simplifies programming but may introduce overhead.

Allocate UM:

float *x;
cudaMallocManaged(&x, N * sizeof(float));

You can then use x on both host and device without explicit cudaMemcpy. To ensure data is ready on the device, you can prefetch:

cudaMemPrefetchAsync(x, N * sizeof(float), deviceId, 0);

UM is great for prototyping and for algorithms with irregular data access. For performance‑critical code, manual transfers are still recommended.

2.8 Multi‑GPU Programming

Modern systems may have multiple GPUs. You can use them to scale your application.

Use cudaGetDeviceCount to find number of devices.
Set current device with cudaSetDevice(deviceId) before allocating memory or launching kernels.
Each device has its own memory space; you must manage data distribution and communication between devices (e.g., via PCIe or NVLink). Use peer‑to‑peer access if available (cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess).

Simple multi‑GPU example:

int numDevices;
cudaGetDeviceCount(&numDevices);

#pragma omp parallel for
for (int dev = 0; dev < numDevices; ++dev) {
    cudaSetDevice(dev);
    // Allocate memory and launch kernel on this device
    // Copy data to/from host (strided or partitioned)
}

2.9 Advanced Example: Parallel Reduction

Reduction (summing an array) is a common pattern. We'll implement an optimized version that uses shared memory and avoids warp divergence.

__global__ void reduce(float *g_idata, float *g_odata, int n) {
    extern __shared__ float sdata[];  // Dynamic shared memory

    unsigned int tid = threadIdx.x;
    unsigned int i = blockIdx.x * blockDim.x + threadIdx.x;

    // Load data into shared memory (with bounds check)
    sdata[tid] = (i < n) ? g_idata[i] : 0.0f;
    __syncthreads();

    // Perform reduction in shared memory
    for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
        if (tid < s) {
            sdata[tid] += sdata[tid + s];
        }
        __syncthreads();
    }

    // Write result for this block to global memory
    if (tid == 0) {
        g_odata[blockIdx.x] = sdata[0];
    }
}

Launch:

int threads = 256;
int blocks = (n + threads - 1) / threads;
reduce<<<blocks, threads, threads * sizeof(float)>>>(d_in, d_out, n);

After this kernel, you have partial sums per block. You may need a second kernel to combine them, or run a single block on the CPU.

Part 3: Introduction to HIP

3.1 What is HIP? Why Portability?

HIP (Heterogeneous‑compute Interface for Portability) is an AMD initiative to provide a C++ runtime API that can target both NVIDIA and AMD GPUs. It is essentially a thin abstraction layer: HIP code compiles to CUDA (using a CUDA backend) when targeting NVIDIA, and to AMD’s ROCm when targeting AMD. This means you can write your kernel once and run it everywhere.

Major benefits:

Vendor independence: No lock‑in to NVIDIA.
Single source: One codebase for multiple platforms.
Ease of porting: CUDA code can be mechanically converted using hipify tools.

3.2 Setting Up HIP

On AMD systems with ROCm:

Follow the ROCm installation guide. On Ubuntu:

  sudo apt update
  sudo apt install rocm-dev

Verify with hipcc --version.

On NVIDIA systems (HIP on CUDA):

Install the CUDA Toolkit first.
Then install HIP from ROCm’s GitHub or via package manager. Many Linux distributions provide a hip-cuda package.
After installation, hipcc will use the CUDA backend.

Environment:

Ensure hipcc is in your PATH.
For AMD, you may need to set ROCM_PATH if not default.

3.3 HIP API Overview

HIP intentionally mimics CUDA. Most CUDA functions have direct HIP equivalents with the hip prefix:

CUDA	HIP
`cudaMalloc`	`hipMalloc`
`cudaFree`	`hipFree`
`cudaMemcpy`	`hipMemcpy`
`cudaMemcpyAsync`	`hipMemcpyAsync`
`cudaDeviceSynchronize`	`hipDeviceSynchronize`
`cudaStreamCreate`	`hipStreamCreate`
`cudaEventCreate`	`hipEventCreate`
`cudaGetLastError`	`hipGetLastError`
`cudaError_t`	`hipError_t`
`cudaSuccess`	`hipSuccess`

The kernel language is essentially the same: __global__, __device__, __shared__, threadIdx, blockIdx, etc., are all supported.

The only major syntactic difference is the kernel launch. Instead of triple chevrons, HIP uses a function macro hipLaunchKernelGGL:

hipLaunchKernelGGL(kernel_name, dim3(grid), dim3(block), sharedMem, stream, args...);

If you prefer the <<<>>> syntax, you can define a macro (but it's not provided by default).

3.4 Porting a CUDA Program to HIP

Let’s port our vector addition from CUDA to HIP.

Original CUDA kernel (unchanged):

__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        c[i] = a[i] + b[i];
    }
}

Host code converted to HIP:

#include <iostream>
#include <hip/hip_runtime.h>

int main() {
    int n = 1 << 20;
    size_t bytes = n * sizeof(float);

    float *h_a = new float[n];
    float *h_b = new float[n];
    float *h_c = new float[n];

    for (int i = 0; i < n; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    float *d_a, *d_b, *d_c;
    hipMalloc(&d_a, bytes);
    hipMalloc(&d_b, bytes);
    hipMalloc(&d_c, bytes);

    hipMemcpy(d_a, h_a, bytes, hipMemcpyHostToDevice);
    hipMemcpy(d_b, h_b, bytes, hipMemcpyHostToDevice);

    int threadsPerBlock = 256;
    int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;

    // HIP kernel launch
    hipLaunchKernelGGL(vecAdd, dim3(blocksPerGrid), dim3(threadsPerBlock), 0, 0, d_a, d_b, d_c, n);

    hipMemcpy(h_c, d_c, bytes, hipMemcpyDeviceToHost);

    // Verification (omitted for brevity)

    hipFree(d_a); hipFree(d_b); hipFree(d_c);
    delete[] h_a; delete[] h_b; delete[] h_c;

    return 0;
}

Compile with:

hipcc -o vecAdd vecAdd.cpp   # on AMD
# or on NVIDIA
hipcc --offload-arch=sm_70 -o vecAdd vecAdd.cpp

Notice that the kernel itself is identical to the CUDA version – that’s the beauty of HIP!

3.5 HIP Kernel Launch Syntax Explained

hipLaunchKernelGGL takes the following arguments:

Kernel name (symbol)
Grid dimensions (dim3)
Block dimensions (dim3)
Dynamic shared memory size per block (size_t, default 0)
Stream (hipStream_t, default 0 for null stream)
Kernel arguments (comma‑separated)

If you have many arguments, it can become verbose. Some projects define a macro like:

#define HIP_LAUNCH(kernel, grid, block, shared, stream, ...) \
    hipLaunchKernelGGL(kernel, grid, block, shared, stream, __VA_ARGS__)

3.6 Error Handling in HIP

HIP error handling is identical to CUDA:

hipError_t err = hipMalloc(&d_a, bytes);
if (err != hipSuccess) {
    std::cerr << "HIP error: " << hipGetErrorString(err) << std::endl;
    exit(EXIT_FAILURE);
}

After a kernel launch, you can check for launch errors with hipGetLastError() and for execution errors after synchronization with hipDeviceSynchronize().

3.7 Debugging and Profiling HIP

Debugging:

On AMD, use rocgdb (similar to cuda‑gdb). Compile with -g and run rocgdb ./vecAdd.
On NVIDIA, you can still use cuda-gdb because the HIP code is translated to CUDA under the hood.

Profiling:

On AMD: Use rocprof to collect performance counters and traces.

  rocprof --hip-trace ./vecAdd

This generates a results.json file that can be viewed with tools like perfetto.

On NVIDIA: Use nvprof or nsight-compute as usual.

Memory checking:

On AMD, use compute-sanitizer (if available) or ROCm’s hip-memcheck.

3.8 HIP Examples: Matrix Multiplication

Our tiled matrix multiplication kernel works unchanged in HIP. Simply change the header and launch syntax. Let's see:

#include <hip/hip_runtime.h>
#include <iostream>

#define TILE_SIZE 16

__global__ void matMulShared(const float *A, const float *B, float *C, int N) {
    __shared__ float As[TILE_SIZE][TILE_SIZE];
    __shared__ float Bs[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;
    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;
    for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
        As[ty][tx] = A[row * N + (tile * TILE_SIZE + tx)];
        Bs[ty][tx] = B[(tile * TILE_SIZE + ty) * N + col];
        __syncthreads();

        for (int k = 0; k < TILE_SIZE; ++k) {
            sum += As[ty][k] * Bs[k][tx];
        }
        __syncthreads();
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

int main() {
    int N = 1024;
    size_t bytes = N * N * sizeof(float);

    // ... host allocations and initialization ...

    float *d_A, *d_B, *d_C;
    hipMalloc(&d_A, bytes);
    hipMalloc(&d_B, bytes);
    hipMalloc(&d_C, bytes);

    hipMemcpy(d_A, h_A, bytes, hipMemcpyHostToDevice);
    hipMemcpy(d_B, h_B, bytes, hipMemcpyHostToDevice);

    dim3 block(TILE_SIZE, TILE_SIZE);
    dim3 grid((N + block.x - 1) / block.x, (N + block.y - 1) / block.y);

    hipLaunchKernelGGL(matMulShared, grid, block, 0, 0, d_A, d_B, d_C, N);

    hipMemcpy(h_C, d_C, bytes, hipMemcpyDeviceToHost);

    // ... verification and cleanup ...

    return 0;
}

Compile with hipcc matmul.cpp -o matmul.

Part 4: Advanced HIP and Porting

4.1 Subtle Differences Between CUDA and HIP

While HIP aims for source compatibility, there are some differences to be aware of:

Warp vs Wavefront: NVIDIA warps are 32 threads; AMD wavefronts are 64 (on current architectures). Code that assumes a warp size of 32 (e.g., warp shuffle intrinsics) may need adjustment. HIP provides __hip_warp_size which returns the actual warp/wavefront size (32 on NVIDIA, 64 on AMD). Use it to write portable code.
Intrinsics: Some CUDA intrinsics (e.g., __popc, __ballot_sync) have HIP equivalents (__popc, __ballot_sync work, but check the HIP documentation for availability). For warp‑level primitives, prefer the portable __sync_warp and __all, __any.
Math functions: Most math functions (sin, exp, etc.) are identical. However, double‑precision performance may vary between vendors.
Architecture‑specific features: Features like tensor cores (NVIDIA) or matrix cores (AMD) are not portable. If you need them, you may have to write vendor‑specific code guarded by macros.

4.2 Using hipify Tools

AMD provides tools to automatically convert CUDA source to HIP:

hipify-perl: A Perl script that performs text‑based replacement.

  hipify-perl vecAdd.cu > vecAdd.cpp

It handles most API renames and some launch syntax, but you'll likely need to manually adjust the kernel launches (the triple chevrons are converted to hipLaunchKernelGGL).

hipify-clang: A more robust tool based on Clang’s AST. It can handle complex code and provides better accuracy.

  hipify-clang vecAdd.cu --cuda-path=/usr/local/cuda

It outputs HIP code to stdout or a file.

After conversion, review the output, especially for any use of CUDA‑specific features that don’t have direct HIP equivalents.

4.3 Writing Portable Code: Best Practices

To ensure your code runs efficiently on both platforms:

Use hip runtime API for memory management, streams, events.
Avoid hard‑coding warp size: Use __hip_warp_size or compute dynamically.
Use portable intrinsics: Prefer __syncthreads, __threadfence, __syncwarp (though __syncwarp may need a mask; HIP provides __syncwarp(mask) with similar semantics).
Conditional compilation for vendor‑specific optimizations:

  #ifdef __HIP_PLATFORM_NVIDIA__
      // NVIDIA-specific code (e.g., using CUDA tensor cores)
  #elif __HIP_PLATFORM_AMD__
      // AMD-specific code
  #endif

Profile on both platforms to identify bottlenecks. What's fast on NVIDIA may not be on AMD due to different cache hierarchies, wavefront size, etc. Tune block sizes and tile sizes accordingly.

4.4 Performance Optimization on AMD GPUs

When targeting AMD, use ROCm tools:

rocprof: For profiling.
rocm-smi: To monitor GPU utilization, clock speeds, temperature.
OmniTrace: For tracing API calls and kernel execution.

Key optimization considerations for AMD:

Wavefront size: 64 threads. Your block size should be a multiple of 64 to avoid idle threads.
Shared memory bank conflicts: AMD’s shared memory has 32 banks (like NVIDIA) but the mapping may differ. Experiment with padding to reduce conflicts.
Memory coalescing: Still crucial; consecutive threads should access consecutive memory.
Occupancy: Similar concepts apply, but the register file and shared memory sizes differ. Use hipOccupancyMaxPotentialBlockSize to get recommendations.

4.5 Real-world Example: Sobel Edge Detection (Portable)

Let's combine everything into a real‑world kernel: Sobel edge detection on an 8‑bit grayscale image. We'll write it portably for HIP/CUDA.

#include <hip/hip_runtime.h>  // or <cuda_runtime.h> for CUDA

#define TILE_SIZE 16
#define MASK_SIZE 3

__global__ void sobel(const unsigned char *in, unsigned char *out, int width, int height) {
    // Shared memory for tile + halo (1 pixel halo on each side)
    __shared__ unsigned char tile[TILE_SIZE + 2][TILE_SIZE + 2];

    int x = blockIdx.x * TILE_SIZE + threadIdx.x;
    int y = blockIdx.y * TILE_SIZE + threadIdx.y;

    // Global coordinates including halo
    int gx = x - 1;
    int gy = y - 1;

    // Load tile with boundary check
    if (gx >= 0 && gx < width && gy >= 0 && gy < height)
        tile[threadIdx.y + 1][threadIdx.x + 1] = in[gy * width + gx];
    else
        tile[threadIdx.y + 1][threadIdx.x + 1] = 0;

    // Load halo (simplified: we rely on neighboring threads)
    // For a full implementation, you'd also load the border regions.
    __syncthreads();

    // Compute Sobel only for inner tile pixels (not halo)
    if (threadIdx.x < TILE_SIZE && threadIdx.y < TILE_SIZE && x < width && y < height) {
        int Gx = -tile[threadIdx.y][threadIdx.x] + tile[threadIdx.y][threadIdx.x + 2]
                 -2*tile[threadIdx.y+1][threadIdx.x] + 2*tile[threadIdx.y+1][threadIdx.x+2]
                 -tile[threadIdx.y+2][threadIdx.x] + tile[threadIdx.y+2][threadIdx.x+2];
        int Gy = -tile[threadIdx.y][threadIdx.x] -2*tile[threadIdx.y][threadIdx.x+1] - tile[threadIdx.y][threadIdx.x+2]
                 +tile[threadIdx.y+2][threadIdx.x] +2*tile[threadIdx.y+2][threadIdx.x+1] + tile[threadIdx.y+2][threadIdx.x+2];
        int mag = min(255, (int)hypot(Gx, Gy));
        out[y * width + x] = mag;
    }
}

Launch:

dim3 block(TILE_SIZE, TILE_SIZE);
dim3 grid((width + block.x - 1) / block.x, (height + block.y - 1) / block.y);
sobel<<<grid, block>>>(d_in, d_out, width, height); // CUDA
// HIP: hipLaunchKernelGGL(sobel, grid, block, 0, 0, d_in, d_out, width, height);

This kernel uses shared memory to reduce global memory traffic. Each block loads a tile plus halo, then computes Sobel for its interior. It's a great example of a real‑world, portable GPU kernel.

Part 5: Conclusion and Next Steps

Congratulations! You've journeyed from the absolute basics of GPU programming to writing advanced, portable kernels that run on both NVIDIA and AMD hardware. You've learned:

The CUDA programming model, memory hierarchy, and essential APIs.
How to write and optimize kernels for maximum performance.
The HIP ecosystem and how to write truly portable code.
Debugging and profiling techniques for both platforms.

Where to Go from Here

Explore official documentation:
- CUDA C++ Programming Guide
- HIP Documentation
Study advanced examples: Reduction, scan, stencils, FFTs.
Contribute to open‑source projects that use CUDA/HIP (e.g., PyTorch, TensorFlow, ROCm libraries).
Experiment with multi‑GPU and distributed computing (e.g., using MPI with GPUs).
Learn about new hardware features like tensor cores (NVIDIA) and matrix cores (AMD) – but remember to keep your code portable or use conditional compilation.

GPU programming is a powerful skill that opens doors to high‑performance computing, machine learning, and graphics. With CUDA and HIP under your belt, you're well‑equipped to tackle the most demanding computational challenges. Now go forth and write some fast, portable code!

If you found this guide helpful, please share it with your fellow developers. Have questions or want to share your own experiences? Drop a comment below – I’d love to hear from you. Happy coding!

OLSRT: The Runtime I Built on Windows 7 When I Couldn't Run MSVC (And Why It Might Be the Last Runtime You'll Ever Need)

Javad — Sat, 14 Feb 2026 16:04:54 +0000

Hey Dev Community!

A while back, I was sitting on my old Windows 7 machine, trying to build a PHP extension. I wanted to bring modern async I/O, WebSocket support, and an event loop to PHP — something like what Node.js has, but for PHP. The problem? phpize requires MSVC, and on Windows 7 with MinGW, I couldn't make it work. No matter what I tried, MSVC just wouldn't install.

So I sat there, frustrated, staring at my screen. And then a thought hit me:

"C can talk to any language. So why am I limiting myself to just one?"

That question changed everything. Instead of building a PHP extension, I started building something much bigger: a language-agnostic runtime that any language could talk to. A runtime that would give every language access to async I/O, WebSocket, event loops, and more.

That was the birth of OLSRT.

From 0.1 to v1.2 (Atom)

The first version (0.1) was a mess. Just a single src folder, no header files, and barely any structure. But it worked — enough to prove the concept.

Then I rewrote everything from scratch. Today, OLSRT is at v1.2 (Atom) , and it's production-ready. Here's what's in it right now:

✅ What OLSRT v1.2 Offers Today

1. Actor Model with Full Process Isolation

Each actor runs in its own isolated process with a private memory arena. If one actor crashes, the rest keep running — just like Erlang/OTP, but in C. The implementation includes:

Lock‑free mailboxes with zero‑copy message passing
Supervisors with restart strategies (permanent, transient, temporary)
Process linking and monitoring (Erlang‑style)

2. Green Threads (Not Just Coroutines)

OLSRT has a full green thread implementation with:

Work stealing across multiple CPU cores
NUMA‑aware memory allocation
SIMD‑accelerated memory operations (memcpy, memset, memcmp)
Guard pages to detect stack overflows
Context switching in assembly for x86_64, ARM64, ARM, and RISC‑V

3. High‑Performance Memory Management

Arena‑based memory allocators (2MB default, expandable)
Zero‑copy IPC between processes
Lock‑free data structures for hot paths
Cache‑line padding to prevent false sharing

4. Async I/O and Event Loop

Cross‑platform event loop with epoll (Linux), kqueue (macOS/BSD), and select (fallback)
Timer support with deadlines
I/O event registration with callbacks

5. Channels and Streams

Thread‑safe FIFO channels with bounded/unbounded capacity
Reactive streams with operators (map, filter, take, merge, debounce)
Backpressure support with demand‑based flow control

6. Promises and Futures

Thread‑safe promise/future implementation
Continuations (.then())
Integration with the event loop for waking

7. Synchronization Primitives

Mutexes, condition variables, read‑write locks, semaphores
All implemented natively on Windows, Linux, macOS, and BSD

8. Networking (TCP/UDP)

Non‑blocking TCP and UDP sockets
Full integration with the event loop
Promise‑based async API (send/recv/accept/connect)

🚧 What's Coming in v1.3 (In Development Now)

Based on our latest commits (as of February 2026), v1.3 will include:

Hot‑Coding References (HCR) – Never pause the main with very lightweight branches.
OLSRT Coroutines (ORoutines) – Even lighter‑weight concurrency
Fully documented runtime
Optimized build systems (CMake and Makefile improvements)

Why OLSRT Exists (The Philosophy)

I didn't set out to build a universal runtime. I just wanted to make PHP async. But along the way, I realized something:

The future of software is polyglot. We use PHP for the web, Python for data, Rust for systems, and JavaScript for the frontend. But today, each language has its own runtime, its own concurrency model, and its own set of limitations. OLSRT is an attempt to unify them — to give every language access to the same powerful primitives: actors, green threads, zero‑copy IPC, and fault‑tolerant supervisors.

Where We Stand Today

OLSRT v1.2 is stable and production‑ready on Linux and BSD (Windows support is coming in v1.3). It has:

118 commits
Full Doxygen documentation (for v1.0, v1.2 docs are in progress)
A growing set of bindings in the works (PHP, JavaScript, Python, Rust, Go, .NET, Dart, Java, Kotlin, D, Elixir, Erlang, Ruby…)
A small but passionate team of 1 core developer ( And it's me :) )

And yes, right now we have exactly 3 GitHub stars. That's it.

Why I'm Writing This

Because OLSRT needs you. Not just as a user, but as a contributor.

This project has the potential to change how we build software — but it can't do that alone. We need:

Developers to star the repo (it sounds small, but it makes a huge difference in visibility)
People to open issues (every bug report, every feature request, every question)
Contributors to send pull requests — even if it's just fixing a typo in the docs
Community members to spread the word (share this post, tweet about OLSRT, talk about it at meetups)

How You Can Help Right Now

Visit the repository: github.com/OverLab-Group/OLSRT
Star it. One click. It takes 2 seconds.
Read the code. It's well‑documented (I promise).
Open an issue. Found a bug? Have an idea? Let me know.
Submit a PR. Even a tiny one counts. Fix a typo, improve a comment, add a test.
Tell a friend. Share this post on Twitter, LinkedIn, or your local dev community.

The Road Ahead

We have a roadmap to v12. But we won't get there alone. OLSRT is open source, and it belongs to the community. With your help, it can become the universal runtime that powers the next generation of software — from real‑time chat servers to telecom infrastructure to embedded IoT devices.

It all started on a Windows 7 machine with a broken MSVC install. Now it's up to you to help it grow.

Join us on GitHub

🚀 OLSRT v1.2: A Powerful Runtime for All Programming Languages!

Javad — Sun, 08 Feb 2026 23:10:18 +0000

Hey Dev Community!

I'm thrilled to be writing blogs for you again after a while!

📖 Introduction

OLSRT (OverLab Streams Runtime) is a powerful, minimal runtime (~90 KB) delivered as shared libraries (.so for Linux/BSD) and dynamic libraries (.dll for NT), written in C, and designed to power all programming languages!

If your language can talk to C (yes, even COBOL!), you can use OLSRT directly via FFI.

✅ Supported Languages: All of them! (JavaScript (with nodeffi), PHP (with C extensions), Python (with cffi), Rust, D, C++, Go, Dart, Kotlin, Java, C#, Julia, and any language you use!)

📌 Note for C, C++, D, Rust, and other languages with C at their core:

You can use OLSRT directly in your programs—even without any .so/.dll files! Just include the header for your required module and use OLSRT with zero bridging.

🔥 What’s New in Version 1.2?

This release brings exciting and advanced features to OLSRT:

🎭 Improved Actors & Supervisors

The first C-based Actors designed for all programming languages! OLSRT now delivers an advanced Actor model for concurrent systems.

⚡ Enhanced Async I/O, Await, Promises & Futures

Full support for modern concurrency patterns across all languages.

🧵 Green Threads & Modern Fibers

Now you can leverage OLSRT's powerful Green Threads in all your projects.

✨ Many Other Improvements

Check out the full release notes for OLSRT Atom v1.2.

📘 Documentation for v1.2 is still in progress, but you can refer to the v1.0 docs (the core concepts remain similar). We’re actively working on examples and comprehensive documentation.

🚧 In Development: OLSRT Version 1.3

We’re already working on v1.3 with revolutionary features:

🧩 Fully Isolated Actors

💥 “Let It Crash!” Philosophy

🔄 ORoutines (OLSRT’s Modern Coroutines)

🧠 HCR (Hot-Coding References) – A Brand New Paradigm!

🤔 What is HCR? A New Paradigm!

OverLab Group divides concurrency paradigms into two categories:

🕐 Waiting Paradigms	⚡ Non-Waiting Paradigms
Blocking:	Branch-based:
– Sync	– HCR (First of its kind!)
– Lock
– Semaphore
Non-Blocking:
– Async
– Await
– Actors

🧪 HCR in Action:

In waiting paradigms, you write code like this:

int x = a + b;

This pauses the main program for milliseconds.

But with HCR:

ol_caller* x = ol_reference_do(a + b);

Here, OLSRT creates a lightweight branch (~1 KB) for x, a, and b, processes them without stopping the main program, and merges the result back seamlessly!

📢 Call to Action

If you like the OLSRT project and want to give us some energy, even doing one of the following helps a ton:

⭐ Star the repository (One energy)
🔀 Fork the repository (One super energy)
💬 Comment, share your ideas, and follow us (One mega energy)
🗣️ Discuss with us on GitHub (Three mega energies)
🛠️ Clone and use the project (One ultra energy)
🐛 Open an issue with your ideas (Three ultra energies)
🤝 Contribute to the repository (One mega ultra super energy)

🔗 GitHub Repository:

https://github.com/OverLab-Group/OLSRT

We welcome all forms of feedback and contribution!

Hope you enjoyed this update.

Have an ORoutine day! 🚀

Stranger Things #1: VHDL wrotes GTN (Guess The Number)?? WTF!

Javad — Wed, 07 Jan 2026 00:53:30 +0000

Hello, hardware enthusiasts and developers! Today, we’re diving into a project that blends the rigorous world of hardware design with a dash of whimsical humor. Welcome to GTN (Guess The Number), a strange VHDL project that will keep you guessing—literally! Built with VHDL, one of the most industrial and challenging Hardware Description Languages (HDLs), GTN combines seriousness with fun, resulting in a quirky system that plays mind games with you. Let’s explore this "Stranger Things" style project!

What is GTN?

GTN is a hardware system that, when initialized, displays a number and asks you to guess what that number was. But here’s the twist:

If you guess correctly, it says, "No! It was wrong," shows you a false number (just to mess with you), and restarts.
If you guess incorrectly, it says, "No, it was wrong," and displays the real number it originally showed, adding a bit of pressure (LOL).

It’s all in good fun, designed to poke fun at perfectionism in hardware design and bring a smile to your face. Now, let’s get into the technical details!

VHDL Source Code

We’ll provide two versions of the code: one for emulation (simulation) and one for FPGA/ASIC boards. The core logic is similar, but I/O handling differs based on the target.

Version 1: Emulation Version

This version is designed for simulation in tools like ModelSim or GHDL. It uses standard logic signals for input and output. Below is the VHDL code for the entity and architecture.

-- GTN_Emulate.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_Emulate is
    Port (
        clk            : in  STD_LOGIC;
        reset          : in  STD_LOGIC;
        guess          : in  STD_LOGIC_VECTOR (7 downto 0);  -- 8-bit guess input
        number_display : out STD_LOGIC_VECTOR (7 downto 0);  -- 8-bit number display
        response       : out STD_LOGIC_VECTOR (1 downto 0)   -- "00": init, "01": correct guess (false response), "10": incorrect guess
    );
end GTN_Emulate;

architecture Behavioral of GTN_Emulate is
    signal internal_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal displayed_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal state : integer range 0 to 2 := 0;  -- 0: init, 1: wait for guess, 2: respond
    signal fake_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
begin
    process (clk, reset)
        variable seed : integer := 123456789;  -- Simple seed for pseudo-random number
    begin
        if reset = '1' then
            internal_number   <= (others => '0');
            displayed_number  <= (others => '0');
            fake_number       <= (others => '0');
            state             <= 0;
            response          <= "00";
        elsif rising_edge(clk) then
            case state is
                when 0 =>
                    -- Initialize: generate a "random" number
                    seed := seed + 1;
                    internal_number  <= std_logic_vector(to_unsigned(seed mod 256, 8));
                    displayed_number <= internal_number;  -- Display real number
                    response         <= "00";              -- Init state
                    state            <= 1;

                when 1 =>
                    -- Wait for guess input (assume guess is stable in simulation)
                    if guess = internal_number then
                        -- Correct guess: show fake number (real number + 1)
                        fake_number       <= std_logic_vector(unsigned(internal_number) + 1);
                        displayed_number  <= fake_number;
                        response          <= "01";          -- Correct guess, but false response
                    else
                        -- Incorrect guess: show real number
                        displayed_number  <= internal_number;
                        response          <= "10";          -- Incorrect guess
                    end if;
                    state <= 2;

                when 2 =>
                    -- Respond and restart after one clock cycle
                    state <= 0;
            end case;
        end if;
    end process;

    number_display <= displayed_number;
end Behavioral;

Testbench for Emulation

To simulate the design, here’s a simple testbench:

-- GTN_Emulate_tb.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_Emulate_tb is
end GTN_Emulate_tb;

architecture sim of GTN_Emulate_tb is
    signal clk            : STD_LOGIC := '0';
    signal reset          : STD_LOGIC := '0';
    signal guess          : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal number_display : STD_LOGIC_VECTOR (7 downto 0);
    signal response       : STD_LOGIC_VECTOR (1 downto 0);

    constant clk_period : time := 10 ns;
begin
    uut: entity work.GTN_Emulate
        port map (
            clk            => clk,
            reset          => reset,
            guess          => guess,
            number_display => number_display,
            response       => response
        );

    clk_process: process
    begin
        clk <= '0';
        wait for clk_period/2;
        clk <= '1';
        wait for clk_period/2;
    end process;

    stimulus: process
begin
    -- Reset the system
    reset <= '1';
    wait for 20 ns;
    reset <= '0';
    wait for 20 ns;

    -- Let system initialize and display a number
    wait until rising_edge(clk);
    wait for clk_period;

    -- Guess correctly (match the displayed number)
    guess <= number_display;
    wait for clk_period * 2;
    report "No!! Your wrong! The number was = " &
           integer'image(to_integer(unsigned(number_display))) &
           " But you wrote = " &
           integer'image(to_integer(unsigned(guess)));

    -- Guess incorrectly (add 5 to the displayed number)
    guess <= std_logic_vector(unsigned(number_display) + 5);
    wait for clk_period * 2;
    report "You wrote wrong again! The number was = " &
           integer'image(to_integer(unsigned(number_display))) &
           " but you wrote = " &
           integer'image(to_integer(unsigned(guess)));

    wait for 100 ns;
    assert false report "Simulation finished. Are you really think you can win? LOL" severity failure;
end process;
end sim;

Just change the fake_number <= std_logic_vector(unsigned(internal_number) + YOUR_NUMBER_HERE); in GTN_Emulate, then you can see the result! (Because of limitions of HDL Languages (Not only VHDL, all of them), we can't input straight to user, and the users must change it from source code, but in real FPGA/ASIC Boards are easier!)

Version 2: FPGA/ASIC Board Version

This version is tailored for physical implementation, with inputs from switches and buttons, and outputs to LEDs. It includes button debouncing for reliability.

-- GTN_FPGA.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_FPGA is
    Port (
        clk            : in  STD_LOGIC;                     -- Board clock (e.g., 100 MHz)
        reset          : in  STD_LOGIC;                     -- Reset button
        guess_switches : in  STD_LOGIC_VECTOR (7 downto 0); -- 8 switches for guess input
        submit_button  : in  STD_LOGIC;                     -- Button to submit guess
        leds           : out STD_LOGIC_VECTOR (7 downto 0); -- 8 LEDs to display number
        response_led   : out STD_LOGIC_VECTOR (1 downto 0)  -- Additional LEDs for response feedback
    );
end GTN_FPGA;

architecture Behavioral of GTN_FPGA is
    signal internal_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal displayed_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal state : integer range 0 to 2 := 0;
    signal button_debounced : STD_LOGIC := '0';
    signal fake_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
begin
    -- Simple button debouncing circuit
    process (clk)
        variable count : integer := 0;
        constant debounce_limit : integer := 1000000; -- Adjust based on clock frequency (e.g., 10 ms debounce)
    begin
        if rising_edge(clk) then
            if submit_button = '1' then
                if count < debounce_limit then
                    count := count + 1;
                else
                    button_debounced <= '1';
                end if;
            else
                count := 0;
                button_debounced <= '0';
            end if;
        end if;
    end process;

    process (clk, reset)
        variable seed : integer := 123456789;
    begin
        if reset = '1' then
            internal_number   <= (others => '0');
            displayed_number  <= (others => '0');
            fake_number       <= (others => '0');
            state             <= 0;
            response_led      <= "00";
        elsif rising_edge(clk) then
            case state is
                when 0 =>
                    -- Initialize and display a number
                    seed := seed + 1;
                    internal_number  <= std_logic_vector(to_unsigned(seed mod 256, 8));
                    displayed_number <= internal_number;
                    response_led     <= "00";  -- Init state
                    state            <= 1;

                when 1 =>
                    -- Wait for guess submission
                    if button_debounced = '1' then
                        if guess_switches = internal_number then
                            -- Correct guess: show fake number
                            fake_number       <= std_logic_vector(unsigned(internal_number) + 1);
                            displayed_number  <= fake_number;
                            response_led      <= "01";  -- Correct guess, false response
                        else
                            -- Incorrect guess: show real number
                            displayed_number  <= internal_number;
                            response_led      <= "10";  -- Incorrect guess
                        end if;
                        state <= 2;
                    end if;

                when 2 =>
                    -- Hold response until reset or auto-restart after delay
                    -- For simplicity, we wait for reset. You can add a timer here.
                    null;
            end case;
        end if;
    end process;

    leds <= displayed_number;
end Behavioral;

Tutorial: Emulation and Synthesis

Emulation (Simulation)

Install a VHDL simulator: Use GHDL (open-source) or ModelSim (commercial). For GHDL, download from ghdl.free.fr.
Save the code: Create two files: GTN_Emulate.vhd and GTN_Emulate_tb.vhd.
Compile and run (with GHDL):

   ghdl -a GTN_Emulate.vhd
   ghdl -a GTN_Emulate_tb.vhd
   ghdl -e GTN_Emulate_tb
   ghdl -r GTN_Emulate_tb --vcd=gtn.vcd

View waveforms: Use GTKWave to open gtn.vcd and analyze signals like number_display, guess, and response.

Synthesis for FPGA

Choose your toolchain: Use Xilinx Vivado for Xilinx FPGAs or Intel Quartus for Intel FPGAs.
Create a new project: Add the GTN_FPGA.vhd file.
Assign pins: Map the ports to your board’s physical pins (e.g., clocks, buttons, switches, LEDs). Refer to your board’s schematic.
Generate bitstream: Run synthesis, implementation, and generate the programming file.
Program the FPGA: Connect your board, load the bitstream, and test!
- After reset, LEDs show a number.
- Set guess using switches, press submit button.
- LEDs update based on your guess—remember the GTN twist!

Bonus Challenge

Think you can beat GTN?

If you do, we’ll hand over the keys to AWS in your name (LOL).

Spoiler: Nobody ever wins. 😉

Call to Action

Got a strange hardware project idea? Built something quirky with VHDL, Verilog, or any HDL? Share it with us! We’d love to hear about it and help broadcast it to the world. Let’s celebrate the fun side of hardware design together.

Reach out to me on Telegram: JavadInteger. Send a message with your project details, and we might feature it in our next blog post!

Thanks for reading, and happy coding! May your guesses always be wrong—or are they? 😉

Distributed Systems & Networking: Full Complete Guide for Beginners!

Javad — Wed, 07 Jan 2026 00:26:47 +0000

Hey Dev Community!
Welcome!

Introduction: Why Theory Matters in Distributed Systems

Distributed systems are the backbone of modern computing, but they're notoriously complex. The difference between systems that work and systems that scale reliably often comes down to understanding the theoretical foundations. This guide will take you from zero to expert, covering many of critical theoretical concepts with complete, production-ready implementations in Python, Go, and C++.

Prerequisites: Basic programming knowledge. No distributed systems experience required. We'll build everything from scratch.

1. TCP Throughput Models: Padhye Model from First Principles

The Complete Mathematical Foundation

TCP throughput isn't just "bandwidth/RTT." The Padhye model provides a complete analytical solution that considers:

Slow start phase: Exponential growth until threshold
Congestion avoidance: Additive Increase Multiplicative Decrease (AIMD)
Timeout and loss recovery: Retransmission dynamics

Complete derivation:

Let's derive the Padhye model step by step:

Throughput = (Packets per cycle) / (Time per cycle)

Where a cycle is:
1. Successfully transmit W packets (window size)
2. Experience a loss at packet W+1
3. Recover via timeout or fast retransmit

The Padhye equation for TCP Reno:

T(p) = min(W_max/RTT, 
          1 / [RTT * √(2bp/3) + T_0 * min(1, 3√(3bp/8)) * p(1+32p²)])

Where:

p = packet loss probability
b = packets acknowledged per ACK (typically 2)
T_0 = timeout duration
W_max = maximum congestion window

Complete Python Implementation with Network Simulation

import numpy as np
import matplotlib.pyplot as plt
from dataclasses import dataclass
from typing import List, Tuple, Optional
import heapq
import random
from enum import Enum

class TCPState(Enum):
    SLOW_START = 1
    CONGESTION_AVOIDANCE = 2
    FAST_RECOVERY = 3
    RETRANSMIT_TIMEOUT = 4

@dataclass
class Packet:
    seq_num: int
    sent_time: float
    acked: bool = False
    lost: bool = False
    retrans_count: int = 0

@dataclass
class NetworkEvent:
    time: float
    event_type: str  # 'packet_sent', 'packet_lost', 'ack_received', 'timeout'
    packet: Optional[Packet] = None
    cwnd: Optional[float] = None
    ssthresh: Optional[float] = None

class TCPSimulation:
    """
    Complete TCP Reno implementation with exact Padhye model validation
    """

    def __init__(self, 
                 RTT: float = 0.1,
                 packet_loss_prob: float = 0.01,
                 mss: int = 1460,  # Maximum Segment Size
                 init_cwnd: int = 10,  # Initial congestion window
                 buffer_size: int = 100):

        # Network parameters
        self.RTT = RTT
        self.packet_loss_prob = packet_loss_prob
        self.mss = mss
        self.buffer_size = buffer_size

        # TCP state variables
        self.cwnd = float(init_cwnd)  # Congestion window (packets)
        self.ssthresh = float('inf')  # Slow start threshold
        self.dup_ack_count = 0
        self.recover_seq = None

        # Sequence tracking
        self.next_seq = 0
        self.last_acked = -1
        self.max_seq_sent = -1

        # Timing and events
        self.current_time = 0.0
        self.event_queue = []  # Min-heap for events
        self.packets_in_flight = []
        self.packets_waiting_ack = {}

        # Statistics
        self.total_packets_sent = 0
        self.total_packets_lost = 0
        self.total_bytes_transferred = 0
        self.start_time = None

        # Event history for analysis
        self.history: List[NetworkEvent] = []

    def _schedule_event(self, delay: float, event_type: str, packet: Packet = None):
        """Schedule a future event using min-heap"""
        event_time = self.current_time + delay
        heapq.heappush(self.event_queue, (event_time, event_type, packet))

    def send_packet(self):
        """Send a new packet if window allows"""
        if len(self.packets_waiting_ack) >= self.cwnd:
            return  # Window full

        packet = Packet(seq_num=self.next_seq, sent_time=self.current_time)
        self.next_seq += 1
        self.max_seq_sent = max(self.max_seq_sent, packet.seq_num)

        # Schedule potential loss and ACK
        if random.random() < self.packet_loss_prob:
            # Packet lost
            packet.lost = True
            self.total_packets_lost += 1
            self._schedule_event(self.RTT, 'packet_lost', packet)
        else:
            # Packet sent successfully, schedule ACK
            self.packets_waiting_ack[packet.seq_num] = packet
            self._schedule_event(self.RTT, 'ack_received', packet)

        self.total_packets_sent += 1
        self.total_bytes_transferred += self.mss

        # Record event
        self.history.append(NetworkEvent(
            time=self.current_time,
            event_type='packet_sent',
            packet=packet,
            cwnd=self.cwnd,
            ssthresh=self.ssthresh
        ))

        return packet

    def handle_ack(self, packet: Packet):
        """Handle ACK reception with Reno congestion control"""

        if packet.seq_num <= self.last_acked:
            # Duplicate ACK
            self.dup_ack_count += 1

            if self.dup_ack_count == 3:
                # Fast retransmit
                self.ssthresh = max(self.cwnd / 2, 2)
                self.cwnd = self.ssthresh + 3  # Window inflation
                self.recover_seq = self.max_seq_sent

                # Resend lost packet
                self._retransmit_packet(self.last_acked + 1)

            elif self.dup_ack_count > 3:
                # Additional duplicate ACKs in fast recovery
                self.cwnd += 1  # Window inflation

        else:
            # New ACK
            self.dup_ack_count = 0
            self.last_acked = packet.seq_num

            # Remove ACKed packets from waiting list
            seqs_to_remove = [s for s in self.packets_waiting_ack 
                             if s <= packet.seq_num]
            for seq in seqs_to_remove:
                del self.packets_waiting_ack[seq]

            # Congestion control
            if self.cwnd < self.ssthresh:
                # Slow start: exponential growth
                self.cwnd += 1
            else:
                # Congestion avoidance: additive growth
                self.cwnd += 1 / self.cwnd

            # Reset recovery state if needed
            if (self.recover_seq is not None and 
                packet.seq_num > self.recover_seq):
                self.cwnd = self.ssthresh
                self.recover_seq = None

    def _retransmit_packet(self, seq_num: int):
        """Retransmit a specific packet"""
        if seq_num in self.packets_waiting_ack:
            packet = self.packets_waiting_ack[seq_num]
            packet.retrans_count += 1
            packet.sent_time = self.current_time

            # Schedule ACK with probability (1 - loss)
            if random.random() > self.packet_loss_prob:
                self._schedule_event(self.RTT, 'ack_received', packet)

    def run(self, duration: float = 10.0):
        """Run simulation for specified duration"""
        self.start_time = self.current_time

        # Initial packet burst
        for _ in range(int(self.cwnd)):
            self.send_packet()

        # Main event loop
        while self.current_time - self.start_time < duration and self.event_queue:
            event_time, event_type, packet = heapq.heappop(self.event_queue)
            self.current_time = event_time

            if event_type == 'ack_received':
                self.handle_ack(packet)
                # Send new packets if window allows
                while len(self.packets_waiting_ack) < self.cwnd:
                    self.send_packet()

            elif event_type == 'packet_lost':
                # Handle timeout
                self.ssthresh = max(self.cwnd / 2, 2)
                self.cwnd = 1  # Back to slow start
                self.dup_ack_count = 0

                # Resend all unacked packets
                for seq in list(self.packets_waiting_ack.keys()):
                    self._retransmit_packet(seq)

    def calculate_throughput(self) -> Tuple[float, float]:
        """Calculate actual and theoretical throughput"""
        duration = self.current_time - self.start_time

        # Actual throughput from simulation
        actual_throughput = self.total_bytes_transferred / duration  # bytes/sec

        # Theoretical throughput from Padhye model
        # Constants for TCP Reno
        b = 2  # packets per ACK
        T0 = 1.0  # timeout in seconds

        p = self.packet_loss_prob
        RTT = self.RTT

        # Padhye equation
        sqrt_term = np.sqrt(2 * b * p / 3)
        timeout_term = T0 * min(1, 3 * np.sqrt(3 * b * p / 8))
        theoretical_packets = 1 / (RTT * sqrt_term + timeout_term * p * (1 + 32 * p**2))

        # Convert to bytes/sec
        theoretical_throughput = theoretical_packets * self.mss

        return actual_throughput, theoretical_throughput

    def plot_results(self):
        """Visualize TCP behavior"""
        times = [e.time for e in self.history]
        cwnds = [e.cwnd for e in self.history]
        events = [e.event_type for e in self.history]

        fig, axes = plt.subplots(2, 2, figsize=(12, 8))

        # Plot 1: Congestion window over time
        ax1 = axes[0, 0]
        ax1.plot(times, cwnds, 'b-', linewidth=1)
        ax1.set_xlabel('Time (seconds)')
        ax1.set_ylabel('Congestion Window (packets)')
        ax1.set_title('TCP Reno Congestion Window')
        ax1.grid(True, alpha=0.3)

        # Plot 2: Event types
        ax2 = axes[0, 1]
        event_counts = {'packet_sent': 0, 'loss': 0}
        for event in self.history:
            if 'sent' in event.event_type:
                event_counts['packet_sent'] += 1
            elif 'lost' in event.event_type:
                event_counts['loss'] += 1

        ax2.bar(event_counts.keys(), event_counts.values())
        ax2.set_ylabel('Count')
        ax2.set_title('Event Distribution')

        # Plot 3: Throughput comparison
        ax3 = axes[1, 0]
        actual, theoretical = self.calculate_throughput()
        labels = ['Simulation', 'Padhye Model']
        values = [actual / 1e6, theoretical / 1e6]  # Convert to Mbps

        ax3.bar(labels, values)
        ax3.set_ylabel('Throughput (Mbps)')
        ax3.set_title(f'Throughput Comparison (Loss: {self.packet_loss_prob*100:.1f}%)')
        ax3.text(0, values[0], f'{values[0]:.2f}', ha='center', va='bottom')
        ax3.text(1, values[1], f'{values[1]:.2f}', ha='center', va='bottom')

        # Plot 4: Window size distribution
        ax4 = axes[1, 1]
        ax4.hist(cwnds, bins=20, alpha=0.7, edgecolor='black')
        ax4.set_xlabel('Window Size')
        ax4.set_ylabel('Frequency')
        ax4.set_title('Congestion Window Distribution')

        plt.tight_layout()
        plt.show()

        # Print statistics
        print("\n" + "="*60)
        print("TCP SIMULATION RESULTS")
        print("="*60)
        print(f"Duration: {self.current_time:.2f} seconds")
        print(f"Packets sent: {self.total_packets_sent}")
        print(f"Packets lost: {self.total_packets_lost} ({self.total_packets_lost/self.total_packets_sent*100:.1f}%)")
        print(f"Actual throughput: {actual/1e6:.2f} Mbps")
        print(f"Theoretical (Padhye): {theoretical/1e6:.2f} Mbps")
        print(f"Difference: {abs(actual-theoretical)/theoretical*100:.1f}%")

# Run comprehensive analysis
def analyze_tcp_behavior():
    """Complete analysis of TCP under different conditions"""

    print("="*60)
    print("COMPREHENSIVE TCP THROUGHPUT ANALYSIS")
    print("="*60)

    # Test different loss rates
    loss_rates = [0.001, 0.005, 0.01, 0.02, 0.05]
    results = []

    for loss_rate in loss_rates:
        print(f"\nTesting with packet loss: {loss_rate*100:.1f}%")

        sim = TCPSimulation(RTT=0.1, packet_loss_prob=loss_rate)
        sim.run(duration=30.0)
        actual, theoretical = sim.calculate_throughput()

        results.append({
            'loss_rate': loss_rate,
            'actual_mbps': actual / 1e6,
            'theoretical_mbps': theoretical / 1e6,
            'packets_sent': sim.total_packets_sent,
            'packets_lost': sim.total_packets_lost
        })

        # Show one detailed plot
        if loss_rate == 0.01:
            sim.plot_results()

    # Plot throughput vs loss rate
    fig, ax = plt.subplots(figsize=(10, 6))

    loss_rates = [r['loss_rate'] for r in results]
    actuals = [r['actual_mbps'] for r in results]
    theoreticals = [r['theoretical_mbps'] for r in results]

    ax.loglog(loss_rates, actuals, 'bo-', label='Simulation', linewidth=2, markersize=8)
    ax.loglog(loss_rates, theoreticals, 'r--', label='Padhye Model', linewidth=2)

    ax.set_xlabel('Packet Loss Rate (log scale)')
    ax.set_ylabel('Throughput (Mbps, log scale)')
    ax.set_title('TCP Throughput vs Packet Loss Rate')
    ax.legend()
    ax.grid(True, which='both', alpha=0.3)

    plt.show()

    # Print summary table
    print("\n" + "="*60)
    print("SUMMARY TABLE")
    print("="*60)
    print(f"{'Loss Rate':<10} {'Actual (Mbps)':<15} {'Theoretical (Mbps)':<18} {'Difference':<10}")
    print("-"*60)

    for r in results:
        diff = abs(r['actual_mbps'] - r['theoretical_mbps']) / r['theoretical_mbps'] * 100
        print(f"{r['loss_rate']*100:>6.2f}% {r['actual_mbps']:>14.2f} {r['theoretical_mbps']:>17.2f} {diff:>9.1f}%")

# Run the complete analysis
if __name__ == "__main__":
    analyze_tcp_behavior()

Complete Go Implementation: Production-Grade TCP Analyzer

package main

import (
    "container/heap"
    "fmt"
    "math"
    "math/rand"
    "sort"
    "time"
)

// Packet represents a network packet with TCP semantics
type Packet struct {
    SeqNum       int
    SentTime     float64
    Acked        bool
    Lost         bool
    RetransCount int
}

// TCPState represents TCP connection state
type TCPState int

const (
    SlowStart TCPState = iota
    CongestionAvoidance
    FastRecovery
    RetransmitTimeout
)

// TCPConnection represents a complete TCP Reno implementation
type TCPConnection struct {
    // Configuration
    rtt             float64
    lossProb        float64
    mss             int
    initCwnd        float64
    bufferSize      int

    // State variables
    state           TCPState
    cwnd            float64
    ssthresh        float64
    dupAckCount     int
    recoverSeq      int

    // Sequence tracking
    nextSeq         int
    lastAcked       int
    maxSeqSent      int

    // Timing
    currentTime     float64
    packetsInFlight []*Packet
    packetsWaiting  map[int]*Packet

    // Statistics
    totalSent       int
    totalLost       int
    totalBytes      int64
    startTime       float64

    // Event queue (min-heap)
    events          EventQueue
}

// Event represents a network or timer event
type Event struct {
    time      float64
    eventType string
    packet    *Packet
}

// EventQueue implements heap.Interface for events
type EventQueue []Event

func (eq EventQueue) Len() int           { return len(eq) }
func (eq EventQueue) Less(i, j int) bool { return eq[i].time < eq[j].time }
func (eq EventQueue) Swap(i, j int)      { eq[i], eq[j] = eq[j], eq[i] }

func (eq *EventQueue) Push(x interface{}) {
    *eq = append(*eq, x.(Event))
}

func (eq *EventQueue) Pop() interface{} {
    old := *eq
    n := len(old)
    item := old[n-1]
    *eq = old[0 : n-1]
    return item
}

// NewTCPConnection creates a new TCP connection with given parameters
func NewTCPConnection(rtt, lossProb float64) *TCPConnection {
    return &TCPConnection{
        rtt:            rtt,
        lossProb:       lossProb,
        mss:            1460,
        initCwnd:       10.0,
        bufferSize:     100,
        cwnd:           10.0,
        ssthresh:       math.Inf(1),
        packetsWaiting: make(map[int]*Packet),
        events:         make(EventQueue, 0),
    }
}

func (tcp *TCPConnection) scheduleEvent(delay float64, eventType string, pkt *Packet) {
    heap.Push(&tcp.events, Event{
        time:      tcp.currentTime + delay,
        eventType: eventType,
        packet:    pkt,
    })
}

func (tcp *TCPConnection) sendPacket() *Packet {
    // Check if window is full
    if len(tcp.packetsWaiting) >= int(tcp.cwnd) {
        return nil
    }

    pkt := &Packet{
        SeqNum:   tcp.nextSeq,
        SentTime: tcp.currentTime,
    }
    tcp.nextSeq++
    if tcp.nextSeq > tcp.maxSeqSent {
        tcp.maxSeqSent = tcp.nextSeq
    }

    // Determine if packet is lost
    if rand.Float64() < tcp.lossProb {
        pkt.Lost = true
        tcp.totalLost++
        tcp.scheduleEvent(tcp.rtt, "packet_lost", pkt)
    } else {
        tcp.packetsWaiting[pkt.SeqNum] = pkt
        tcp.scheduleEvent(tcp.rtt, "ack_received", pkt)
    }

    tcp.totalSent++
    tcp.totalBytes += int64(tcp.mss)

    return pkt
}

func (tcp *TCPConnection) handleAck(pkt *Packet) {
    if pkt.SeqNum <= tcp.lastAcked {
        // Duplicate ACK
        tcp.dupAckCount++

        if tcp.dupAckCount == 3 {
            // Fast retransmit
            tcp.ssthresh = math.Max(tcp.cwnd/2, 2)
            tcp.cwnd = tcp.ssthresh + 3
            tcp.recoverSeq = tcp.maxSeqSent
            tcp.state = FastRecovery

            // Retransmit the lost packet
            tcp.retransmitPacket(tcp.lastAcked + 1)
        } else if tcp.dupAckCount > 3 && tcp.state == FastRecovery {
            // Additional duplicate ACKs in fast recovery
            tcp.cwnd++
        }
    } else {
        // New ACK
        tcp.dupAckCount = 0
        tcp.lastAcked = pkt.SeqNum

        // Remove ACKed packets
        for seq := range tcp.packetsWaiting {
            if seq <= pkt.SeqNum {
                delete(tcp.packetsWaiting, seq)
            }
        }

        // Update congestion window based on state
        switch tcp.state {
        case SlowStart:
            tcp.cwnd++
            if tcp.cwnd >= tcp.ssthresh {
                tcp.state = CongestionAvoidance
            }
        case CongestionAvoidance:
            tcp.cwnd += 1.0 / tcp.cwnd
        case FastRecovery:
            tcp.cwnd = tcp.ssthresh
            tcp.state = CongestionAvoidance
            tcp.recoverSeq = -1
        }
    }
}

func (tcp *TCPConnection) retransmitPacket(seqNum int) {
    if pkt, exists := tcp.packetsWaiting[seqNum]; exists {
        pkt.RetransCount++
        pkt.SentTime = tcp.currentTime

        if rand.Float64() > tcp.lossProb {
            tcp.scheduleEvent(tcp.rtt, "ack_received", pkt)
        }
    }
}

func (tcp *TCPConnection) Run(duration float64) {
    tcp.startTime = tcp.currentTime

    // Send initial burst of packets
    for i := 0; i < int(tcp.cwnd); i++ {
        tcp.sendPacket()
    }

    // Main event loop
    for tcp.currentTime-tcp.startTime < duration && len(tcp.events) > 0 {
        event := heap.Pop(&tcp.events).(Event)
        tcp.currentTime = event.time

        switch event.eventType {
        case "ack_received":
            tcp.handleAck(event.packet)
            // Send more packets if window allows
            for len(tcp.packetsWaiting) < int(tcp.cwnd) {
                if tcp.sendPacket() == nil {
                    break
                }
            }
        case "packet_lost":
            // Timeout
            tcp.ssthresh = math.Max(tcp.cwnd/2, 2)
            tcp.cwnd = 1
            tcp.dupAckCount = 0
            tcp.state = SlowStart

            // Retransmit all unacked packets
            for seq := range tcp.packetsWaiting {
                tcp.retransmitPacket(seq)
            }
        }
    }
}

// CalculateThroughput computes actual and theoretical throughput
func (tcp *TCPConnection) CalculateThroughput() (float64, float64) {
    duration := tcp.currentTime - tcp.startTime
    if duration == 0 {
        return 0, 0
    }

    // Actual throughput
    actual := float64(tcp.totalBytes) / duration

    // Theoretical throughput using Padhye model
    b := 2.0
    T0 := 1.0
    p := tcp.lossProb
    RTT := tcp.rtt

    sqrtTerm := math.Sqrt(2 * b * p / 3)
    timeoutFactor := 3 * math.Sqrt(3*b*p/8)
    if timeoutFactor > 1 {
        timeoutFactor = 1
    }
    timeoutTerm := T0 * timeoutFactor

    theoreticalPackets := 1.0 / (RTT*sqrtTerm + timeoutTerm*p*(1+32*p*p))
    theoretical := theoreticalPackets * float64(tcp.mss)

    return actual, theoretical
}

// PadhyeThroughput calculates throughput directly from Padhye equation
func PadhyeThroughput(RTT, p float64) float64 {
    b := 2.0
    T0 := 1.0

    if p == 0 {
        return math.Inf(1)
    }

    sqrtTerm := math.Sqrt(2 * b * p / 3)
    timeoutFactor := 3 * math.Sqrt(3*b*p/8)
    if timeoutFactor > 1 {
        timeoutFactor = 1
    }
    timeoutTerm := T0 * timeoutFactor

    return 1.0 / (RTT*sqrtTerm + timeoutTerm*p*(1+32*p*p)) * 1460
}

func main() {
    rand.Seed(time.Now().UnixNano())

    fmt.Println("="*60)
    fmt.Println("GO TCP THROUGHPUT ANALYZER")
    fmt.Println("="*60)

    // Test different network conditions
    testCases := []struct {
        name   string
        rtt    float64
        loss   float64
        duration float64
    }{
        {"Good Network", 0.05, 0.001, 20},
        {"Average Network", 0.1, 0.01, 20},
        {"Poor Network", 0.2, 0.05, 20},
        {"Satellite Link", 0.5, 0.02, 30},
    }

    for _, tc := range testCases {
        fmt.Printf("\nTesting: %s\n", tc.name)
        fmt.Printf("RTT: %.3fs, Loss: %.3f%%\n", tc.rtt, tc.loss*100)

        tcp := NewTCPConnection(tc.rtt, tc.loss)
        tcp.Run(tc.duration)

        actual, theoretical := tcp.CalculateThroughput()

        fmt.Printf("Actual throughput: %.2f Mbps\n", actual/1e6)
        fmt.Printf("Padhye prediction: %.2f Mbps\n", theoretical/1e6)
        fmt.Printf("Difference: %.1f%%\n", 
            math.Abs(actual-theoretical)/theoretical*100)
    }

    // Analyze BDP and Bufferbloat
    fmt.Println("\n" + "="*60)
    fmt.Println("BANDWIDTH-DELAY PRODUCT ANALYSIS")
    fmt.Println("="*60)

    analyzeBDP()
}

func analyzeBDP() {
    // Calculate BDP for different scenarios
    scenarios := []struct {
        bandwidth float64 // Mbps
        rtt       float64 // ms
    }{
        {100, 10},   // LAN
        {50, 50},    // Broadband
        {10, 100},   // Cellular
        {1, 200},    // Rural
    }

    fmt.Println("\nBDP Calculation:")
    fmt.Printf("%-15s %-10s %-10s %-15s %-20s\n", 
        "Scenario", "BW (Mbps)", "RTT (ms)", "BDP (KB)", "Optimal Buffer")

    for _, s := range scenarios {
        // BDP in bits
        bdpBits := s.bandwidth * 1e6 * (s.rtt / 1000)

        // Convert to bytes
        bdpBytes := bdpBits / 8

        // Optimal buffer (1x BDP)
        optimalBuffer := bdpBytes

        // Typical oversized buffer (100x BDP)
        oversizedBuffer := bdpBytes * 100

        // Queuing delay with oversized buffer at 50% utilization
        queuingDelay := (oversizedBuffer * 0.5) / (s.bandwidth * 1e6 / 8)

        fmt.Printf("%-15s %-10.0f %-10.0f %-15.0f %.0fKB (%.0fms delay)\n",
            "", s.bandwidth, s.rtt, bdpBytes/1024,
            oversizedBuffer/1024, queuingDelay*1000)
    }

    // Bufferbloat simulation
    fmt.Println("\nBufferbloat Simulation:")
    simulateBufferbloat()
}

func simulateBufferbloat() {
    // Simulate the effect of oversized buffers
    const (
        bandwidth   = 100e6  // 100 Mbps
        RTT         = 0.05   // 50ms
        packetSize  = 1500   // bytes
    )

    bdp := bandwidth * RTT / 8  // Bytes in flight

    fmt.Printf("\nBandwidth: %.0f Mbps\n", bandwidth/1e6)
    fmt.Printf("RTT: %.0f ms\n", RTT*1000)
    fmt.Printf("BDP: %.2f KB\n", bdp/1024)

    // Test different buffer sizes
    bufferMultipliers := []float64{1, 10, 50, 100, 200}

    fmt.Printf("\n%-15s %-15s %-20s %-25s\n", 
        "Buffer Size", "Multiplier", "Queuing Delay", "Effective RTT")

    for _, mult := range bufferMultipliers {
        bufferSize := bdp * mult

        // Queuing delay at 80% utilization
        queueDelay := (bufferSize * 0.8) / (bandwidth / 8)

        // Effective RTT (base + queuing)
        effectiveRTT := RTT + queueDelay

        // TCP throughput with bufferbloat
        throughput := PadhyeThroughput(effectiveRTT, 0.001)

        fmt.Printf("%-15.0fKB %-15.0fx %-20.0fms %-25.0fms\n",
            bufferSize/1024, mult, queueDelay*1000, effectiveRTT*1000)
    }
}

2. CRDTs: Complete Mathematical Foundation and Implementation

The Complete Theory of CRDTs

CRDTs are based on mathematical monoids and lattice theory. Let's build from first principles:

Definition: A CRDT is a data type whose operations commute:

∀ operations a,b: a ∘ b ≡ b ∘ a

Mathematical Foundations:

Semilattice: A set S with a join operation ⊔ that is:
- Commutative: x ⊔ y = y ⊔ x
- Associative: (x ⊔ y) ⊔ z = x ⊔ (y ⊔ z)
- Idempotent: x ⊔ x = x
Monotonic growth: State can only grow according to a partial order:

   x ≤ y iff x ⊔ y = y

Complete C++ Implementation: Production-Ready CRDT Library

#include <iostream>
#include <vector>
#include <map>
#include <set>
#include <unordered_map>
#include <unordered_set>
#include <memory>
#include <algorithm>
#include <cassert>
#include <cmath>
#include <mutex>
#include <sstream>
#include <type_traits>

// ============================================================================
// LATTICE THEORY IMPLEMENTATION
// ============================================================================

template<typename T>
class Lattice {
public:
    virtual T join(const T& other) const = 0;
    virtual bool lessEqual(const T& other) const = 0;
    virtual std::string toString() const = 0;
    virtual ~Lattice() = default;

    // Lattice properties verification
    static bool verifyCommutative(const T& a, const T& b) {
        return a.join(b) == b.join(a);
    }

    static bool verifyAssociative(const T& a, const T& b, const T& c) {
        return (a.join(b)).join(c) == a.join(b.join(c));
    }

    static bool verifyIdempotent(const T& a) {
        return a.join(a) == a;
    }
};

// ============================================================================
// VECTOR CLOCKS FOR CAUSAL ORDERING
// ============================================================================

class VectorClock : public Lattice<VectorClock> {
private:
    std::map<std::string, int64_t> clocks_;

public:
    VectorClock() = default;

    explicit VectorClock(const std::string& nodeId) {
        clocks_[nodeId] = 0;
    }

    void increment(const std::string& nodeId) {
        clocks_[nodeId]++;
    }

    int64_t get(const std::string& nodeId) const {
        auto it = clocks_.find(nodeId);
        return it != clocks_.end() ? it->second : 0;
    }

    // Join operation: take maximum of each component
    VectorClock join(const VectorClock& other) const override {
        VectorClock result;

        // Add all from this
        for (const auto& [node, time] : clocks_) {
            result.clocks_[node] = time;
        }

        // Take maximum with other
        for (const auto& [node, time] : other.clocks_) {
            auto it = result.clocks_.find(node);
            if (it == result.clocks_.end() || time > it->second) {
                result.clocks_[node] = time;
            }
        }

        return result;
    }

    // Partial order: less or equal if all components are ≤
    bool lessEqual(const VectorClock& other) const override {
        for (const auto& [node, time] : clocks_) {
            if (time > other.get(node)) {
                return false;
            }
        }

        // Check nodes in other but not in this (implicitly 0 in this)
        for (const auto& [node, time] : other.clocks_) {
            if (get(node) > time) {
                return false;
            }
        }

        return true;
    }

    // Concurrent if neither ≤ other
    bool concurrent(const VectorClock& other) const {
        return !lessEqual(other) && !other.lessEqual(*this);
    }

    bool operator==(const VectorClock& other) const {
        return clocks_ == other.clocks_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "{";
        for (auto it = clocks_.begin(); it != clocks_.end(); ++it) {
            if (it != clocks_.begin()) oss << ", ";
            oss << it->first << ":" << it->second;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// CRDT BASE CLASSES
// ============================================================================

template<typename T>
class CRDT : public Lattice<CRDT<T>> {
public:
    virtual void apply(const T& operation) = 0;
    virtual CRDT<T> merge(const CRDT<T>& other) const = 0;
    virtual T value() const = 0;

    CRDT<T> join(const CRDT<T>& other) const override {
        return this->merge(other);
    }

    bool lessEqual(const CRDT<T>& other) const override {
        // For state-based CRDTs: x ≤ y iff x ⊔ y = y
        auto joined = this->merge(other);
        return joined == other;
    }
};

// ============================================================================
// G-COUNTER (GROW-ONLY COUNTER)
// ============================================================================

class GCounter : public CRDT<GCounter> {
private:
    std::map<std::string, int64_t> counts_;

public:
    GCounter() = default;

    explicit GCounter(const std::string& nodeId) {
        counts_[nodeId] = 0;
    }

    void increment(const std::string& nodeId, int64_t delta = 1) {
        if (delta <= 0) return;
        counts_[nodeId] += delta;
    }

    int64_t value() const override {
        int64_t total = 0;
        for (const auto& [_, count] : counts_) {
            total += count;
        }
        return total;
    }

    int64_t get(const std::string& nodeId) const {
        auto it = counts_.find(nodeId);
        return it != counts_.end() ? it->second : 0;
    }

    GCounter merge(const GCounter& other) const override {
        GCounter result;

        // Start with our counts
        result.counts_ = counts_;

        // Take maximum for each node
        for (const auto& [node, count] : other.counts_) {
            auto it = result.counts_.find(node);
            if (it == result.counts_.end() || count > it->second) {
                result.counts_[node] = count;
            }
        }

        return result;
    }

    void apply(const GCounter& operation) override {
        // For operation-based: increment locally
        // For state-based: merge
        *this = this->merge(operation);
    }

    bool operator==(const GCounter& other) const {
        return counts_ == other.counts_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "GCounter(" << value() << ")[";
        for (auto it = counts_.begin(); it != counts_.end(); ++it) {
            if (it != counts_.begin()) oss << ", ";
            oss << it->first << ":" << it->second;
        }
        oss << "]";
        return oss.str();
    }
};

// ============================================================================
// PN-COUNTER (POSITIVE-NEGATIVE COUNTER)
// ============================================================================

class PNCounter : public CRDT<PNCounter> {
private:
    GCounter increments_;
    GCounter decrements_;

public:
    PNCounter() = default;

    explicit PNCounter(const std::string& nodeId) 
        : increments_(nodeId + "_inc"), decrements_(nodeId + "_dec") {}

    void increment(const std::string& nodeId, int64_t delta = 1) {
        increments_.increment(nodeId, delta);
    }

    void decrement(const std::string& nodeId, int64_t delta = 1) {
        decrements_.increment(nodeId, delta);
    }

    int64_t value() const override {
        return increments_.value() - decrements_.value();
    }

    PNCounter merge(const PNCounter& other) const override {
        PNCounter result;
        result.increments_ = increments_.merge(other.increments_);
        result.decrements_ = decrements_.merge(other.decrements_);
        return result;
    }

    void apply(const PNCounter& operation) override {
        *this = this->merge(operation);
    }

    bool operator==(const PNCounter& other) const {
        return increments_ == other.increments_ && 
               decrements_ == other.decrements_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "PNCounter(" << value() << ")";
        return oss.str();
    }
};

// ============================================================================
// G-SET (GROW-ONLY SET)
// ============================================================================

template<typename T>
class GSet : public CRDT<GSet<T>> {
private:
    std::set<T> elements_;

public:
    void add(const T& element) {
        elements_.insert(element);
    }

    bool contains(const T& element) const {
        return elements_.find(element) != elements_.end();
    }

    const std::set<T>& value() const {
        return elements_;
    }

    GSet<T> merge(const GSet<T>& other) const override {
        GSet<T> result;
        std::set_union(elements_.begin(), elements_.end(),
                      other.elements_.begin(), other.elements_.end(),
                      std::inserter(result.elements_, result.elements_.begin()));
        return result;
    }

    void apply(const GSet<T>& operation) override {
        *this = this->merge(operation);
    }

    bool operator==(const GSet<T>& other) const {
        return elements_ == other.elements_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "GSet{";
        for (auto it = elements_.begin(); it != elements_.end(); ++it) {
            if (it != elements_.begin()) oss << ", ";
            oss << *it;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// 2P-SET (TWO-PHASE SET)
// ============================================================================

template<typename T>
class TwoPSet : public CRDT<TwoPSet<T>> {
private:
    GSet<T> added_;
    GSet<T> removed_;

public:
    void add(const T& element) {
        if (!removed_.contains(element)) {
            added_.add(element);
        }
    }

    void remove(const T& element) {
        if (added_.contains(element)) {
            removed_.add(element);
        }
    }

    bool contains(const T& element) const {
        return added_.contains(element) && !removed_.contains(element);
    }

    TwoPSet<T> merge(const TwoPSet<T>& other) const override {
        TwoPSet<T> result;

        // Merge added sets
        result.added_ = added_.merge(other.added_);

        // Merge removed sets
        result.removed_ = removed_.merge(other.removed_);

        // Remove elements that were added after being removed
        for (const auto& elem : result.removed_.value()) {
            result.added_.add(elem);  // G-Set add is idempotent
        }

        return result;
    }

    void apply(const TwoPSet<T>& operation) override {
        *this = this->merge(operation);
    }

    std::set<T> value() const {
        std::set<T> result;
        for (const auto& elem : added_.value()) {
            if (!removed_.contains(elem)) {
                result.insert(elem);
            }
        }
        return result;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "2P-Set{";
        auto vals = value();
        for (auto it = vals.begin(); it != vals.end(); ++it) {
            if (it != vals.begin()) oss << ", ";
            oss << *it;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// LWW-REGISTER (LAST-WRITE-WINS REGISTER)
// ============================================================================

template<typename T>
class LWWRegister : public CRDT<LWWRegister<T>> {
private:
    T value_;
    VectorClock timestamp_;
    std::string nodeId_;

public:
    LWWRegister(const std::string& nodeId, const T& initial = T{})
        : value_(initial), nodeId_(nodeId) {
        timestamp_.increment(nodeId);
    }

    void set(const T& value) {
        value_ = value;
        timestamp_.increment(nodeId_);
    }

    T value() const override {
        return value_;
    }

    const VectorClock& timestamp() const {
        return timestamp_;
    }

    LWWRegister<T> merge(const LWWRegister<T>& other) const override {
        // Compare timestamps
        if (timestamp_.concurrent(other.timestamp_)) {
            // Tie-break by nodeId
            if (nodeId_ < other.nodeId_) {
                return *this;
            } else {
                return other;
            }
        } else if (timestamp_.lessEqual(other.timestamp_)) {
            return other;
        } else {
            return *this;
        }
    }

    void apply(const LWWRegister<T>& operation) override {
        *this = this->merge(operation);
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "LWWRegister(" << value_ << ", ts=" << timestamp_.toString() << ")";
        return oss.str();
    }
};

// ============================================================================
// MV-REGISTER (MULTI-VALUE REGISTER)
// ============================================================================

template<typename T>
class MVRegister : public CRDT<MVRegister<T>> {
private:
    std::set<std::pair<T, VectorClock>> versions_;
    std::string nodeId_;

public:
    MVRegister(const std::string& nodeId, const T& initial = T{})
        : nodeId_(nodeId) {
        VectorClock ts;
        ts.increment(nodeId);
        versions_.insert({initial, ts});
    }

    void set(const T& value) {
        // Create new version with incremented timestamp
        VectorClock newTs;

        // Start with maximum of all current versions
        for (const auto& [_, ts] : versions_) {
            newTs = newTs.join(ts);
        }

        // Increment our node's counter
        newTs.increment(nodeId_);

        // Clear old versions and add new one
        versions_.clear();
        versions_.insert({value, newTs});
    }

    std::set<T> value() const {
        std::set<T> result;
        for (const auto& [val, _] : versions_) {
            result.insert(val);
        }
        return result;
    }

    MVRegister<T> merge(const MVRegister<T>& other) const override {
        MVRegister<T> result(nodeId_);
        result.versions_.clear();

        // Collect all versions
        std::set<std::pair<T, VectorClock>> allVersions;
        allVersions.insert(versions_.begin(), versions_.end());
        allVersions.insert(other.versions_.begin(), other.versions_.end());

        // Remove dominated versions
        for (const auto& v1 : allVersions) {
            bool dominated = false;
            for (const auto& v2 : allVersions) {
                if (&v1 == &v2) continue;

                if (v1.second.lessEqual(v2.second)) {
                    dominated = true;
                    break;
                }
            }
            if (!dominated) {
                result.versions_.insert(v1);
            }
        }

        return result;
    }

    void apply(const MVRegister<T>& operation) override {
        *this = this->merge(operation);
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "MVRegister{";
        for (auto it = versions_.begin(); it != versions_.end(); ++it) {
            if (it != versions_.begin()) oss << ", ";
            oss << it->first << ":" << it->second.toString();
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// CRDT REPLICA WITH NETWORK SIMULATION
// ============================================================================

template<typename T>
class CRDTReplica {
private:
    std::string id_;
    std::shared_ptr<CRDT<T>> state_;
    std::vector<std::shared_ptr<CRDTReplica<T>>> peers_;
    std::mutex mtx_;

public:
    CRDTReplica(const std::string& id, std::shared_ptr<CRDT<T>> initialState)
        : id_(id), state_(initialState) {}

    void connect(CRDTReplica<T>* peer) {
        peers_.push_back(std::shared_ptr<CRDTReplica<T>>(peer));
    }

    template<typename Func>
    void applyOperation(Func op) {
        std::lock_guard<std::mutex> lock(mtx_);
        op(state_);
    }

    void gossip() {
        std::lock_guard<std::mutex> lock(mtx_);

        // Send state to random peers
        for (auto& peer : peers_) {
            if (peer) {
                peer->receiveState(state_);
            }
        }
    }

    void receiveState(std::shared_ptr<CRDT<T>> remoteState) {
        std::lock_guard<std::mutex> lock(mtx_);
        auto merged = state_->merge(*remoteState);
        state_ = std::make_shared<CRDT<T>>(merged);
    }

    std::shared_ptr<CRDT<T>> getState() const {
        std::lock_guard<std::mutex> lock(mtx_);
        return state_;
    }

    std::string getId() const { return id_; }
};

// ============================================================================
// DEMONSTRATION AND TESTING
// ============================================================================

void testCRDTProperties() {
    std::cout << "=" << std::string(60, '=') << std::endl;
    std::cout << "CRDT MATHEMATICAL PROPERTIES VERIFICATION" << std::endl;
    std::cout << "=" << std::string(60, '=') << std::endl;

    // Test GCounter
    {
        std::cout << "\n1. Testing GCounter (Grow-Only Counter):\n";

        GCounter c1("node1");
        GCounter c2("node2");
        GCounter c3("node3");

        c1.increment("node1", 5);
        c2.increment("node2", 3);
        c3.increment("node3", 7);

        // Test commutative: c1 ⊔ c2 == c2 ⊔ c1
        auto c1c2 = c1.merge(c2);
        auto c2c1 = c2.merge(c1);

        std::cout << "   Commutative: " 
                  << (c1c2 == c2c1 ? "✓" : "✗") 
                  << " (c1 ⊔ c2 == c2 ⊔ c1)" << std::endl;

        // Test associative: (c1 ⊔ c2) ⊔ c3 == c1 ⊔ (c2 ⊔ c3)
        auto left = c1.merge(c2).merge(c3);
        auto right = c1.merge(c2.merge(c3));

        std::cout << "   Associative: " 
                  << (left == right ? "✓" : "✗")
                  << " ((c1 ⊔ c2) ⊔ c3 == c1 ⊔ (c2 ⊔ c3))" << std::endl;

        // Test idempotent: c1 ⊔ c1 == c1
        auto c1c1 = c1.merge(c1);

        std::cout << "   Idempotent:  "
                  << (c1c1 == c1 ? "✓" : "✗")
                  << " (c1 ⊔ c1 == c1)" << std::endl;

        std::cout << "   Final value: " << left.value() << std::endl;
    }

    // Test PNCounter
    {
        std::cout << "\n2. Testing PNCounter (Positive-Negative Counter):\n";

        PNCounter p1("node1");
        PNCounter p2("node2");

        p1.increment("node1", 10);
        p1.decrement("node1", 3);

        p2.increment("node2", 5);
        p2.decrement("node2", 2);

        auto merged = p1.merge(p2);

        std::cout << "   Node1: " << p1.value() 
                  << " (10 - 3)" << std::endl;
        std::cout << "   Node2: " << p2.value() 
                  << " (5 - 2)" << std::endl;
        std::cout << "   Merged: " << merged.value() 
                  << " (15 - 5)" << std::endl;

        // Test lattice properties
        std::cout << "   Is lattice: " 
                  << (Lattice<PNCounter>::verifyCommutative(p1, p2) ? "✓" : "✗")
                  << std::endl;
    }

    // Test LWW Register
    {
        std::cout << "\n3. Testing LWWRegister (Last-Write-Wins):\n";

        LWWRegister<std::string> r1("node1", "initial");
        LWWRegister<std::string> r2("node2", "initial");

        // Simulate concurrent writes
        r1.set("value from node1");
        r2.set("value from node2");

        auto merged = r1.merge(r2);

        std::cout << "   Node1: " << r1.value() << std::endl;
        std::cout << "   Node2: " << r2.value() << std::endl;
        std::cout << "   Merged: " << merged.value() << std::endl;

        // Check which one won
        if (merged.value() == r1.value()) {
            std::cout << "   Winner: node1 (tie-break by node ID)" << std::endl;
        } else {
            std::cout << "   Winner: node2 (tie-break by node ID)" << std::endl;
        }
    }

    // Test MV Register
    {
        std::cout << "\n4. Testing MVRegister (Multi-Value):\n";

        MVRegister<std::string> mv1("node1", "A");
        MVRegister<std::string> mv2("node2", "B");

        // Concurrent writes
        mv1.set("A1");
        mv2.set("B1");

        auto merged = mv1.merge(mv2);
        auto values = merged.value();

        std::cout << "   After concurrent writes:" << std::endl;
        std::cout << "   Values: {";
        for (const auto& v : values) {
            std::cout << v << " ";
        }
        std::cout << "}" << std::endl;

        // Resolve by writing again
        mv1.set("resolved");
        auto resolved = mv1.merge(mv2);

        std::cout << "   After resolution: " 
                  << *resolved.value().begin() << std::endl;
    }
}

void simulateDistributedCounter() {
    std::cout << "\n" << std::string(60, '=') << std::endl;
    std::cout << "DISTRIBUTED COUNTER SIMULATION" << std::endl;
    std::cout << std::string(60, '=') << std::endl;

    // Create replicas
    auto counter1 = std::make_shared<PNCounter>("node1");
    auto counter2 = std::make_shared<PNCounter>("node2");
    auto counter3 = std::make_shared<PNCounter>("node3");

    CRDTReplica<PNCounter> r1("node1", counter1);
    CRDTReplica<PNCounter> r2("node2", counter2);
    CRDTReplica<PNCounter> r3("node3", counter3);

    // Connect replicas in a network
    r1.connect(&r2);
    r1.connect(&r3);
    r2.connect(&r1);
    r2.connect(&r3);
    r3.connect(&r1);
    r3.connect(&r2);

    // Simulate operations
    std::cout << "\nInitial state:" << std::endl;
    std::cout << "  Node1: " << r1.getState()->value() << std::endl;
    std::cout << "  Node2: " << r2.getState()->value() << std::endl;
    std::cout << "  Node3: " << r3.getState()->value() << std::endl;

    // Each node makes local updates
    r1.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node1", 5);
        pn->decrement("node1", 2);
    });

    r2.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node2", 3);
        pn->decrement("node2", 1);
    });

    r3.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node3", 7);
        pn->decrement("node3", 4);
    });

    std::cout << "\nAfter local updates:" << std::endl;
    std::cout << "  Node1: " << r1.getState()->value() << std::endl;
    std::cout << "  Node2: " << r2.getState()->value() << std::endl;
    std::cout << "  Node3: " << r3.getState()->value() << std::endl;

    // Perform gossip rounds
    for (int round = 1; round <= 3; round++) {
        std::cout << "\nGossip Round " << round << ":" << std::endl;

        r1.gossip();
        r2.gossip();
        r3.gossip();

        std::cout << "  Node1: " << r1.getState()->value() << std::endl;
        std::cout << "  Node2: " << r2.getState()->value() << std::endl;
        std::cout << "  Node3: " << r3.getState()->value() << std::endl;

        // Check for convergence
        auto v1 = r1.getState()->value();
        auto v2 = r2.getState()->value();
        auto v3 = r3.getState()->value();

        if (v1 == v2 && v2 == v3) {
            std::cout << "  ✓ Converged after " << round << " rounds!" << std::endl;
            break;
        }
    }

    // Verify final consistency
    std::cout << "\nFinal verification:" << std::endl;
    std::cout << "  All nodes equal: " 
              << (r1.getState()->value() == r2.getState()->value() && 
                  r2.getState()->value() == r3.getState()->value() ? "✓" : "✗")
              << std::endl;
    std::cout << "  Expected total: 8 (5-2 + 3-1 + 7-4)" << std::endl;
}

void demonstrateCALMTheorem() {
    std::cout << "\n" << std::string(60, '=') << std::endl;
    std::cout << "CALM THEOREM DEMONSTRATION" << std::endl;
    std::cout << "=" << std::string(60, '=') << std::endl;

    // The CALM theorem: Consistency As Logical Monotonicity
    // Monotonic programs don't need coordination

    std::cout << "\nMonotonic vs Non-Monotonic Computations:\n";

    // Example 1: Monotonic query (can use CRDTs)
    std::cout << "\n1. Monotonic: 'Has user visited page X?'\n";
    std::cout << "   - Once true, always true\n";
    std::cout << "   - Can use G-Set (add-only)\n";
    std::cout << "   - No coordination needed ✓\n";

    GSet<std::string> visitedPages;
    visitedPages.add("/home");
    visitedPages.add("/products");

    std::cout << "   Current state: " << visitedPages.toString() << std::endl;

    // Example 2: Non-monotonic query (needs coordination)
    std::cout << "\n2. Non-Monotonic: 'What is the last page user visited?'\n";
    std::cout << "   - Can change over time\n";
    std::cout << "   - Needs LWW-Register or consensus\n";
    std::cout << "   - Coordination required ✗\n";

    LWWRegister<std::string> lastPage("server1", "/home");
    lastPage.set("/products");
    lastPage.set("/checkout");

    std::cout << "   Last page: " << lastPage.value() << std::endl;

    // Mathematical formulation
    std::cout << "\nMathematical Formulation:\n";
    std::cout << "   Monotonic logic: if P ⊢ φ, then P ∪ Q ⊢ φ\n";
    std::cout << "   Non-monotonic:   P ⊢ φ but P ∪ Q ⊬ φ\n";
    std::cout << "\nCALM Theorem: A program has a consistent,\n";
    std::cout << "coordination-free distributed implementation\n";
    std::cout << "iff it is monotonic.\n";
}

int main() {
    std::cout << "COMPLETE CRDT LIBRARY WITH MATHEMATICAL FOUNDATIONS\n";
    std::cout << std::string(60, '=') << std::endl;

    testCRDTProperties();
    simulateDistributedCounter();
    demonstrateCALMTheorem();

    return 0;
}

3. Dynamo-style Quorums: Complete Mathematical Analysis

Complete Theory of Quorum Systems

A quorum system Q is a collection of subsets of nodes where each pair intersects:

∀ Q₁, Q₂ ∈ Q: Q₁ ∩ Q₂ ≠ ∅

Mathematical Properties:

Load: Probability a node is in a randomly chosen quorum
Capacity: Maximum throughput the system can handle
Fault tolerance: Maximum number of failures the system can tolerate

Optimal Quorum Sizes:

For N nodes with read quorum R and write quorum W:

Strong consistency: R + W > N (and W > N/2 to avoid write conflicts)
Eventual consistency: R + W ≤ N

Complete Python Implementation with Proofs

import numpy as np
import itertools
from typing import List, Set, Tuple, Dict
from dataclasses import dataclass
from enum import Enum
import random
import hashlib
from fractions import Fraction
import matplotlib.pyplot as plt

class ConsistencyLevel(Enum):
    STRONG = "strong"
    EVENTUAL = "eventual"
    CAUSAL = "causal"
    LINEARIZABLE = "linearizable"

@dataclass
class QuorumSystem:
    """
    Mathematical analysis of quorum systems
    """
    N: int  # Total nodes
    R: int  # Read quorum size
    W: int  # Write quorum size

    def __post_init__(self):
        if not (1 <= self.R <= self.N and 1 <= self.W <= self.N):
            raise ValueError("Quorum sizes must be between 1 and N")

    @property
    def quorums(self) -> List[Set[int]]:
        """Generate all possible quorums of size R and W"""
        read_quorums = [set(c) for c in itertools.combinations(range(self.N), self.R)]
        write_quorums = [set(c) for c in itertools.combinations(range(self.N), self.W)]
        return read_quorums, write_quorums

    def is_available(self, failed_nodes: Set[int]) -> Tuple[bool, bool]:
        """
        Check if read and write are still possible with failed nodes

        Returns: (read_possible, write_possible)
        """
        # Read is possible if there exists a read quorum without failed nodes
        read_possible = any(failed_nodes.isdisjoint(q) 
                           for q in self.read_quorums)

        # Write is possible if there exists a write quorum without failed nodes
        write_possible = any(failed_nodes.isdisjoint(q) 
                            for q in self.write_quorums)

        return read_possible, write_possible

    @property
    def read_quorums(self) -> List[Set[int]]:
        return [set(c) for c in itertools.combinations(range(self.N), self.R)]

    @property
    def write_quorums(self) -> List[Set[int]]:
        return [set(c) for c in itertools.combinations(range(self.N), self.W)]

    def probability_available(self, p_fail: float) -> Tuple[float, float]:
        """
        Calculate probability that read and write are available

        Args:
            p_fail: Probability a single node fails

        Returns: (P(read available), P(write available))
        """
        # Probability exactly k nodes fail
        def P_k(k: int) -> float:
            return (np.math.comb(self.N, k) * 
                   (p_fail ** k) * 
                   ((1 - p_fail) ** (self.N - k)))

        # Read available if at least R nodes are up
        P_read = sum(P_k(k) for k in range(self.N - self.R + 1))

        # Write available if at least W nodes are up
        P_write = sum(P_k(k) for k in range(self.N - self.W + 1))

        return P_read, P_write

    def consistency_level(self) -> ConsistencyLevel:
        """Determine consistency level based on quorum configuration"""
        if self.R + self.W > self.N and self.W > self.N / 2:
            return ConsistencyLevel.LINEARIZABLE
        elif self.R + self.W > self.N:
            return ConsistencyLevel.STRONG
        elif self.R + self.W == self.N:
            return ConsistencyLevel.CAUSAL
        else:
            return ConsistencyLevel.EVENTUAL

    def load(self) -> Tuple[float, float]:
        """
        Calculate load on nodes for read and write operations

        Returns: (read_load, write_load)
        """
        # Probability a node is in a randomly chosen quorum
        read_load = self.R / self.N
        write_load = self.W / self.N

        return read_load, write_load

    def capacity(self) -> float:
        """
        Calculate maximum throughput capacity

        Returns: Minimum of read and write capacity
        """
        read_load, write_load = self.load()

        # Capacity is inverse of load
        read_capacity = 1 / read_load if read_load > 0 else float('inf')
        write_capacity = 1 / write_load if write_load > 0 else float('inf')

        return min(read_capacity, write_capacity)

    def fault_tolerance(self) -> Dict[str, int]:
        """
        Calculate fault tolerance metrics

        Returns: Dictionary with various tolerance measures
        """
        # Maximum failures before read becomes impossible
        max_failures_read = self.N - self.R

        # Maximum failures before write becomes impossible
        max_failures_write = self.N - self.W

        # Failures that still guarantee at least one intersecting node
        # between any read and write quorum
        if self.R + self.W > self.N:
            max_intersection_failures = self.R + self.W - self.N - 1
        else:
            max_intersection_failures = -1  # No guarantee

        return {
            'max_failures_for_read': max_failures_read,
            'max_failures_for_write': max_failures_write,
            'max_failures_guaranteeing_intersection': max_intersection_failures,
            'can_tolerate_network_partition': self.R + self.W > self.N
        }

    def optimal_configurations(self) -> List['QuorumSystem']:
        """
        Find optimal configurations for given N

        Returns: List of Pareto-optimal configurations
        """
        optimal = []

        for R in range(1, self.N + 1):
            for W in range(1, self.N + 1):
                config = QuorumSystem(self.N, R, W)

                # Skip dominated configurations
                dominated = False
                for opt in optimal[:]:  # Copy list for iteration
                    if (R >= opt.R and W >= opt.W and 
                        config.capacity() <= opt.capacity()):
                        dominated = True
                        break
                    elif (R <= opt.R and W <= opt.W and 
                          config.capacity() >= opt.capacity()):
                        # Current optimal is dominated
                        optimal.remove(opt)

                if not dominated:
                    optimal.append(config)

        return sorted(optimal, key=lambda x: x.capacity(), reverse=True)

class DynamoDBStore:
    """
    Complete Dynamo-style key-value store implementation
    """

    def __init__(self, N: int = 3, R: int = 2, W: int = 2):
        self.quorum_system = QuorumSystem(N, R, W)
        self.nodes = [{} for _ in range(N)]
        self.clocks = [{} for _ in range(N)]  # Vector clocks per key
        self.hinted_handoff = {}  # For handling failures
        self.merkle_trees = [{} for _ in range(N)]  # For anti-entropy

        # Statistics
        self.stats = {
            'reads': 0,
            'writes': 0,
            'read_repairs': 0,
            'hinted_handoffs': 0,
            'conflicts': 0
        }

    def _hash_key(self, key: str) -> List[int]:
        """Consistent hashing to determine node positions"""
        # Using SHA-1 for consistent hashing
        hash_obj = hashlib.sha1(key.encode())
        hash_int = int(hash_obj.hexdigest(), 16)

        # Return N nodes in preference list
        nodes = []
        for i in range(self.quorum_system.N):
            node_idx = (hash_int + i) % self.quorum_system.N
            nodes.append(node_idx)

        return nodes

    def _get_preference_list(self, key: str) -> List[int]:
        """Get ordered list of nodes for a key"""
        return self._hash_key(key)

    def _update_vector_clock(self, node_idx: int, key: str, client_context: Dict = None):
        """Update vector clock for a key"""
        if key not in self.clocks[node_idx]:
            self.clocks[node_idx][key] = {}

        clock = self.clocks[node_idx][key]

        if client_context and 'clock' in client_context:
            # Merge with client's clock
            for node, time in client_context['clock'].items():
                clock[node] = max(clock.get(node, 0), time)

        # Increment this node's counter
        clock[str(node_idx)] = clock.get(str(node_idx), 0) + 1

        return clock

    def _compare_clocks(self, clock1: Dict, clock2: Dict) -> str:
        """
        Compare two vector clocks

        Returns: 'before', 'after', 'concurrent', or 'equal'
        """
        all_nodes = set(clock1.keys()) | set(clock2.keys())

        clock1_before = all(clock1.get(n, 0) <= clock2.get(n, 0) for n in all_nodes)
        clock2_before = all(clock2.get(n, 0) <= clock1.get(n, 0) for n in all_nodes)

        if clock1_before and clock2_before:
            return 'equal'
        elif clock1_before:
            return 'before'
        elif clock2_before:
            return 'after'
        else:
            return 'concurrent'

    def write(self, key: str, value: any, context: Dict = None) -> Dict:
        """
        Write a key-value pair with quorum consistency

        Returns: Context for future reads
        """
        self.stats['writes'] += 1

        preference_list = self._get_preference_list(key)
        successful_writes = 0
        written_nodes = []
        latest_clock = None

        # Try to write to first N nodes in preference list
        for node_idx in preference_list[:self.quorum_system.N]:
            try:
                # Update vector clock
                clock = self._update_vector_clock(node_idx, key, context)
                latest_clock = clock

                # Store value
                self.nodes[node_idx][key] = {
                    'value': value,
                    'clock': clock.copy(),
                    'timestamp': time.time()
                }

                successful_writes += 1
                written_nodes.append(node_idx)

                if successful_writes >= self.quorum_system.W:
                    break  # Quorum achieved

            except Exception as e:
                # Node failed, use hinted handoff
                self._hint_handoff(key, value, clock, node_idx)

        if successful_writes < self.quorum_system.W:
            raise Exception(f"Write failed: only {successful_writes}/{self.quorum_system.W} successful")

        # Return context for client
        return {
            'clock': latest_clock,
            'written_nodes': written_nodes,
            'key': key
        }

    def _hint_handoff(self, key: str, value: any, clock: Dict, failed_node: int):
        """Store data temporarily when primary node is down"""
        self.stats['hinted_handoffs'] += 1

        # Find a healthy node to store hint
        preference_list = self._get_preference_list(key)
        for node_idx in preference_list:
            if node_idx != failed_node and node_idx not in self.hinted_handoff:
                self.hinted_handoff[(key, failed_node)] = {
                    'value': value,
                    'clock': clock,
                    'stored_at': node_idx,
                    'timestamp': time.time()
                }
                break

    def read(self, key: str) -> Tuple[any, Dict]:
        """
        Read a key-value pair with quorum consistency

        Returns: (value, context) or (None, None) if not found
        """
        self.stats['reads'] += 1

        preference_list = self._get_preference_list(key)
        responses = []

        # Read from R nodes
        for node_idx in preference_list[:self.quorum_system.N]:
            if key in self.nodes[node_idx]:
                data = self.nodes[node_idx][key]
                responses.append({
                    'node': node_idx,
                    'value': data['value'],
                    'clock': data['clock'],
                    'timestamp': data['timestamp']
                })

            if len(responses) >= self.quorum_system.R:
                break

        if not responses:
            return None, None

        # Check if we have conflicting versions
        versions = []
        for resp in responses:
            versions.append((resp['value'], resp['clock']))

        # Resolve conflicts
        resolved_value, resolved_clock = self._resolve_conflicts(versions)

        # Perform read repair if needed
        self._read_repair(key, resolved_value, resolved_clock, 
                         [r['node'] for r in responses])

        return resolved_value, {'clock': resolved_clock}

    def _resolve_conflicts(self, versions: List[Tuple[any, Dict]]) -> Tuple[any, Dict]:
        """
        Resolve conflicting versions using vector clocks

        Returns: (resolved_value, merged_clock)
        """
        if len(versions) == 1:
            return versions[0]

        # Check if all versions are causally related
        all_same = all(self._compare_clocks(v1[1], v2[1]) == 'equal' 
                      for v1 in versions for v2 in versions)

        if all_same:
            # No actual conflict
            return versions[0]

        # Find latest version based on vector clocks
        latest = versions[0]
        for version in versions[1:]:
            comparison = self._compare_clocks(latest[1], version[1])
            if comparison == 'before':
                latest = version
            elif comparison == 'concurrent':
                # Concurrent writes: need application-specific resolution
                self.stats['conflicts'] += 1
                # Default: last write wins based on timestamp
                if version[0] > latest[0]:  # Simple heuristic
                    latest = version

        # Merge clocks
        merged_clock = {}
        for value, clock in versions:
            for node, time in clock.items():
                merged_clock[node] = max(merged_clock.get(node, 0), time)

        return latest[0], merged_clock

    def _read_repair(self, key: str, value: any, clock: Dict, read_nodes: List[int]):
        """Repair replicas that have stale data"""
        self.stats['read_repairs'] += 1

        preference_list = self._get_preference_list(key)

        for node_idx in preference_list[:self.quorum_system.N]:
            if node_idx in read_nodes:
                continue

            if key in self.nodes[node_idx]:
                # Compare clocks
                existing_clock = self.nodes[node_idx][key]['clock']
                comparison = self._compare_clocks(existing_clock, clock)

                if comparison == 'before':
                    # Existing data is stale, update it
                    self.nodes[node_idx][key] = {
                        'value': value,
                        'clock': clock,
                        'timestamp': time.time()
                    }
            else:
                # Node doesn't have this key, add it
                self.nodes[node_idx][key] = {
                    'value': value,
                    'clock': clock,
                    'timestamp': time.time()
                }

    def simulate_failures(self, failed_nodes: Set[int]):
        """Simulate node failures"""
        for node_idx in failed_nodes:
            self.nodes[node_idx] = {}  # Clear node data
            print(f"Node {node_idx} failed")

    def get_statistics(self) -> Dict:
        """Get store statistics"""
        # Calculate data distribution
        node_loads = [len(node) for node in self.nodes]
        avg_load = np.mean(node_loads)
        load_std = np.std(node_loads)

        return {
            **self.stats,
            'node_loads': node_loads,
            'avg_load': avg_load,
            'load_std': load_std,
            'load_imbalance': load_std / avg_load if avg_load > 0 else 0,
            'config': {
                'N': self.quorum_system.N,
                'R': self.quorum_system.R,
                'W': self.quorum_system.W,
                'consistency': self.quorum_system.consistency_level().value
            }
        }

def analyze_quorum_tradeoffs_comprehensive():
    """
    Complete analysis of quorum system tradeoffs
    """
    print("="*70)
    print("COMPREHENSIVE QUORUM SYSTEM ANALYSIS")
    print("="*70)

    N = 5  # Number of nodes

    # Analyze all possible configurations
    all_configs = []
    for R in range(1, N + 1):
        for W in range(1, N + 1):
            qs = QuorumSystem(N, R, W)

            # Calculate metrics
            consistency = qs.consistency_level()
            read_load, write_load = qs.load()
            capacity = qs.capacity()
            fault_tolerance = qs.fault_tolerance()
            P_read, P_write = qs.probability_available(0.01)  # 1% failure probability

            all_configs.append({
                'R': R,
                'W': W,
                'consistency': consistency,
                'read_load': read_load,
                'write_load': write_load,
                'capacity': capacity,
                'max_read_failures': fault_tolerance['max_failures_for_read'],
                'max_write_failures': fault_tolerance['max_failures_for_write'],
                'read_availability': P_read,
                'write_availability': P_write,
                'R+W': R + W,
                'R+W>N': R + W > N,
                'W>N/2': W > N / 2
            })

    # Sort by different criteria
    print("\n1. Configurations sorted by consistency strength:")
    print("-"*70)
    sorted_by_consistency = sorted(all_configs, 
                                  key=lambda x: (x['R+W>N'], x['W>N/2']), 
                                  reverse=True)

    for config in sorted_by_consistency[:10]:
        print(f"R={config['R']}, W={config['W']}: "
              f"{config['consistency'].value.upper():12} "
              f"(R+W={config['R+W']}, R+W>N={config['R+W>N']}, W>N/2={config['W>N/2']})")

    print("\n2. Configurations sorted by capacity (throughput):")
    print("-"*70)
    sorted_by_capacity = sorted(all_configs, key=lambda x: x['capacity'], reverse=True)

    for config in sorted_by_capacity[:10]:
        print(f"R={config['R']}, W={config['W']}: "
              f"Capacity={config['capacity']:.2f} ops/sec, "
              f"Availability(R/W)={config['read_availability']:.3f}/{config['write_availability']:.3f}")

    print("\n3. Pareto-optimal configurations (tradeoff frontier):")
    print("-"*70)

    # Find Pareto frontier
    pareto_front = []
    for config in all_configs:
        dominated = False
        for other in all_configs:
            if (other['read_availability'] >= config['read_availability'] and
                other['write_availability'] >= config['write_availability'] and
                other['capacity'] >= config['capacity'] and
                (other['read_availability'] > config['read_availability'] or
                 other['write_availability'] > config['write_availability'] or
                 other['capacity'] > config['capacity'])):
                dominated = True
                break
        if not dominated:
            pareto_front.append(config)

    # Sort by availability
    pareto_front.sort(key=lambda x: x['read_availability'], reverse=True)

    for config in pareto_front:
        print(f"R={config['R']:2d}, W={config['W']:2d}: "
              f"Consistency={config['consistency'].value:12}, "
              f"Avail(R)={config['read_availability']:.3f}, "
              f"Avail(W)={config['write_availability']:.3f}, "
              f"Capacity={config['capacity']:.2f}")

    # Visualize tradeoffs
    visualize_quorum_tradeoffs(all_configs, pareto_front)

def visualize_quorum_tradeoffs(all_configs, pareto_front):
    """Create comprehensive visualizations of quorum tradeoffs"""

    fig, axes = plt.subplots(2, 3, figsize=(15, 10))

    # Plot 1: Availability vs Capacity
    ax1 = axes[0, 0]
    read_avails = [c['read_availability'] for c in all_configs]
    capacities = [c['capacity'] for c in all_configs]
    colors = []

    for config in all_configs:
        if config['consistency'] == ConsistencyLevel.LINEARIZABLE:
            colors.append('red')
        elif config['consistency'] == ConsistencyLevel.STRONG:
            colors.append('orange')
        elif config['consistency'] == ConsistencyLevel.CAUSAL:
            colors.append('yellow')
        else:
            colors.append('blue')

    scatter = ax1.scatter(read_avails, capacities, c=colors, alpha=0.6, s=50)
    ax1.set_xlabel('Read Availability')
    ax1.set_ylabel('Capacity (ops/sec)')
    ax1.set_title('Availability vs Capacity Tradeoff')
    ax1.grid(True, alpha=0.3)

    # Add Pareto frontier
    pareto_avails = [c['read_availability'] for c in pareto_front]
    pareto_caps = [c['capacity'] for c in pareto_front]
    ax1.plot(pareto_avails, pareto_caps, 'k--', alpha=0.5, label='Pareto Frontier')
    ax1.legend()

    # Plot 2: Consistency vs Load
    ax2 = axes[0, 1]
    consistency_levels = [c['consistency'].value for c in all_configs]
    unique_levels = list(set(consistency_levels))
    level_to_num = {level: i for i, level in enumerate(unique_levels)}

    load_imbalance = []
    for config in all_configs:
        # Calculate load imbalance
        imbalance = abs(config['read_load'] - config['write_load'])
        load_imbalance.append(imbalance)

    ax2.scatter([level_to_num[l] for l in consistency_levels], 
                load_imbalance, alpha=0.6)
    ax2.set_xlabel('Consistency Level')
    ax2.set_ylabel('Load Imbalance |R/N - W/N|')
    ax2.set_title('Consistency vs Load Balance')
    ax2.set_xticks(range(len(unique_levels)))
    ax2.set_xticklabels(unique_levels, rotation=45)
    ax2.grid(True, alpha=0.3)

    # Plot 3: Fault Tolerance Heatmap
    ax3 = axes[0, 2]

    # Create heatmap data
    heatmap_data = np.zeros((N, N))
    for config in all_configs:
        R, W = config['R'], config['W']
        # Use availability as metric
        heatmap_data[R-1, W-1] = config['read_availability'] * config['write_availability']

    im = ax3.imshow(heatmap_data, cmap='viridis', origin='lower')
    ax3.set_xlabel('Write Quorum Size (W)')
    ax3.set_ylabel('Read Quorum Size (R)')
    ax3.set_title('Availability Heatmap (R×W)')
    plt.colorbar(im, ax=ax3)

    # Add contour for R+W>N
    X, Y = np.meshgrid(range(1, N+1), range(1, N+1))
    Z = (X + Y) > N
    ax3.contour(X-1, Y-1, Z, levels=[0.5], colors='white', linewidths=2)

    # Add contour for W>N/2
    Z2 = Y > N/2
    ax3.contour(X-1, Y-1, Z2, levels=[0.5], colors='red', linewidths=2, linestyles='--')

    # Plot 4: Capacity vs Consistency
    ax4 = axes[1, 0]

    # Group by consistency level
    grouped_data = {}
    for config in all_configs:
        level = config['consistency'].value
        if level not in grouped_data:
            grouped_data[level] = []
        grouped_data[level].append(config['capacity'])

    boxes = []
    labels = []
    for level, capacities in grouped_data.items():
        boxes.append(capacities)
        labels.append(level)

    ax4.boxplot(boxes, labels=labels)
    ax4.set_ylabel('Capacity (ops/sec)')
    ax4.set_title('Capacity Distribution by Consistency Level')
    ax4.grid(True, alpha=0.3)
    ax4.tick_params(axis='x', rotation=45)

    # Plot 5: Tradeoff Surface (3D)
    ax5 = axes[1, 1]

    # Extract data for 3D plot
    R_vals = [c['R'] for c in all_configs]
    W_vals = [c['W'] for c in all_configs]
    avail_vals = [c['read_availability'] * c['write_availability'] for c in all_configs]
    cap_vals = [c['capacity'] for c in all_configs]

    # Color by consistency
    colors_3d = []
    for config in all_configs:
        if config['consistency'] == ConsistencyLevel.LINEARIZABLE:
            colors_3d.append('red')
        elif config['consistency'] == ConsistencyLevel.STRONG:
            colors_3d.append('orange')
        elif config['consistency'] == ConsistencyLevel.CAUSAL:
            colors_3d.append('green')
        else:
            colors_3d.append('blue')

    scatter = ax5.scatter(R_vals, W_vals, s=np.array(cap_vals)*10, 
                         c=colors_3d, alpha=0.6)
    ax5.set_xlabel('R')
    ax5.set_ylabel('W')
    ax5.set_title('Quorum Configuration Space\n(Size = Capacity)')
    ax5.grid(True, alpha=0.3)

    # Add R+W=N line
    line_x = np.arange(1, N+1)
    line_y = N - line_x
    ax5.plot(line_x, line_y, 'k--', alpha=0.5, label='R+W=N')
    ax5.legend()

    # Plot 6: Failure Tolerance Analysis
    ax6 = axes[1, 2]

    # Simulate different failure scenarios
    failure_rates = np.linspace(0.001, 0.1, 20)

    # Test different configurations
    test_configs = [
        (2, 2),  # Strong
        (1, 3),  # Write-heavy
        (3, 1),  # Read-heavy
        (1, 1),  # Weak
    ]

    for R, W in test_configs:
        qs = QuorumSystem(N, R, W)
        availabilities = []

        for p_fail in failure_rates:
            P_read, P_write = qs.probability_available(p_fail)
            availabilities.append(P_read * P_write)  # Both read and write available

        ax6.plot(failure_rates, availabilities, 
                label=f'R={R}, W={W}', linewidth=2)

    ax6.set_xlabel('Node Failure Probability')
    ax6.set_ylabel('System Availability')
    ax6.set_title('Availability vs Failure Rate')
    ax6.legend()
    ax6.grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()

    # Print optimal configurations for different scenarios
    print("\n4. Recommended configurations for different use cases:")
    print("-"*70)

    recommendations = [
        ("Financial transactions", ConsistencyLevel.LINEARIZABLE, 0.9999, 1000),
        ("Social media feeds", ConsistencyLevel.EVENTUAL, 0.99, 10000),
        ("Shopping cart", ConsistencyLevel.CAUSAL, 0.999, 5000),
        ("Configuration data", ConsistencyLevel.STRONG, 0.9999, 100),
        ("Analytics data", ConsistencyLevel.EVENTUAL, 0.95, 50000),
    ]

    for use_case, required_consistency, required_availability, required_throughput in recommendations:
        print(f"\n{use_case}:")
        print(f"  Requirements: {required_consistency.value}, "
              f"avail>{required_availability}, "
              f"throughput>{required_throughput}")

        # Find matching configurations
        matches = []
        for config in all_configs:
            if (config['consistency'] == required_consistency and
                config['read_availability'] >= required_availability and
                config['capacity'] >= required_throughput):
                matches.append(config)

        if matches:
            # Sort by capacity (ascending for efficiency)
            matches.sort(key=lambda x: x['capacity'])
            best = matches[0]
            print(f"  Recommended: R={best['R']}, W={best['W']}")
            print(f"    Capacity: {best['capacity']:.0f} ops/sec")
            print(f"    Availability: {best['read_availability']:.4f}")
        else:
            print(f"  No configuration meets all requirements")

# Run simulation of Dynamo-style store
def simulate_dynamo_store():
    """Complete simulation of Dynamo-style store with failures"""

    print("\n" + "="*70)
    print("DYNAMO-STORE SIMULATION WITH FAILURES")
    print("="*70)

    # Create store with strong consistency
    store = DynamoDBStore(N=5, R=3, W=3)

    print("\n1. Writing initial data...")
    contexts = {}

    # Write some data
    keys = ["user:1001:profile", "user:1001:cart", "config:feature_flag"]
    values = [{"name": "Alice", "age": 30}, 
              {"items": ["book", "pen"]},
              {"new_ui": True}]

    for key, value in zip(keys, values):
        context = store.write(key, value)
        contexts[key] = context
        print(f"  Wrote {key}: {value}")

    print("\n2. Reading data normally...")
    for key in keys:
        value, ctx = store.read(key)
        print(f"  Read {key}: {value}")

    print("\n3. Simulating network partition...")
    # Partition: nodes 0,1 in one partition, 2,3,4 in another
    store.simulate_failures({0, 1})

    # Try operations during partition
    print("\n   During partition:")

    try:
        # Write during partition (should fail for strong consistency)
        context = store.write("user:1001:profile", 
                            {"name": "Alice", "age": 31},
                            contexts["user:1001:profile"])
        print("   Write succeeded (unexpected!)")
    except Exception as e:
        print(f"   Write failed: {e}")

    try:
        # Read during partition
        value, ctx = store.read("user:1001:profile")
        print(f"   Read succeeded: {value}")
    except Exception as e:
        print(f"   Read failed: {e}")

    print("\n4. Healing partition and testing convergence...")
    # Clear failures
    store.simulate_failures(set())

    # Read after healing
    for key in keys:
        value, ctx = store.read(key)
        print(f"  After healing {key}: {value}")

    print("\n5. Testing concurrent writes...")
    # Simulate concurrent writes from different clients
    print("\n   Client A writes version 1...")
    context_a = store.write("concurrent:key", {"version": 1})

    print("   Client B writes version 2 (without seeing A)...")
    # Create a store that hasn't seen A's write
    store_b = DynamoDBStore(N=5, R=3, W=3)
    context_b = store_b.write("concurrent:key", {"version": 2})

    print("   Merging states...")
    # Now merge the two stores
    for node in range(5):
        if "concurrent:key" in store_b.nodes[node]:
            store.nodes[node]["concurrent:key"] = store_b.nodes[node]["concurrent:key"]

    # Read and resolve conflict
    value, ctx = store.read("concurrent:key")
    print(f"   Resolved value: {value}")

    print("\n6. Statistics:")
    stats = store.get_statistics()
    for key, value in stats.items():
        if key != 'node_loads':
            print(f"  {key}: {value}")

    # Show node load distribution
    print(f"\n  Node loads: {stats['node_loads']}")
    print(f"  Load imbalance: {stats['load_imbalance']:.3f}")

def prove_quorum_properties():
    """
    Mathematical proofs of quorum system properties
    """
    print("\n" + "="*70)
    print("MATHEMATICAL PROOFS OF QUORUM PROPERTIES")
    print("="*70)

    print("\nTheorem 1: For strong consistency, R + W > N")
    print("Proof:")
    print("  1. Let Q_R be a read quorum and Q_W be a write quorum.")
    print("  2. For strong consistency, every read must see the latest write.")
    print("  3. This requires Q_R ∩ Q_W ≠ ∅ (they must intersect).")
    print("  4. By pigeonhole principle, if |Q_R| + |Q_W| > N,")
    print("     then Q_R and Q_W must intersect.")
    print("  5. Therefore, R + W > N is necessary. ✓")

    print("\nTheorem 2: To avoid write conflicts, W > N/2")
    print("Proof:")
    print("  1. Consider two concurrent writes with quorums Q1 and Q2.")
    print("  2. If |Q1| > N/2 and |Q2| > N/2, then Q1 ∩ Q2 ≠ ∅.")
    print("  3. The intersecting node will see both writes and can order them.")
    print("  4. If W ≤ N/2, two writes could use disjoint quorums,")
    print("     leading to conflicts that require complex resolution.")
    print("  5. Therefore, W > N/2 avoids write conflicts. ✓")

    print("\nTheorem 3: Optimal load is achieved when R = W = ⌈(N+1)/2⌉")
    print("Proof:")
    print("  1. Load L = max(R/N, W/N) (bottleneck resource).")
    print("  2. Constraint: R + W > N for strong consistency.")
    print("  3. Minimize L subject to R + W > N, 1 ≤ R,W ≤ N.")
    print("  4. By symmetry and linear programming, minimum occurs when R = W.")
    print("  5. With R = W, constraint becomes 2R > N ⇒ R > N/2.")
    print("  6. Smallest integer satisfying this is R = ⌈(N+1)/2⌉.")
    print("  7. Therefore, R = W = ⌈(N+1)/2⌉ minimizes load. ✓")

    print("\nTheorem 4: Maximum failures tolerated is min(N-R, N-W)")
    print("Proof:")
    print("  1. Read requires at least R functioning nodes.")
    print("  2. Therefore, can tolerate up to N-R failures for read.")
    print("  3. Write requires at least W functioning nodes.")
    print("  4. Therefore, can tolerate up to N-W failures for write.")
    print("  5. System fails if either read or write becomes impossible.")
    print("  6. Therefore, max failures = min(N-R, N-W). ✓")

    print("\nTheorem 5: For N=3, R=2, W=2 is optimal for strong consistency")
    print("Proof:")
    print("  1. Strong consistency requires R + W > 3.")
    print("  2. Possible pairs: (2,2), (1,3), (3,1), (2,3), (3,2).")
    print("  3. Load L = max(R/3, W/3).")
    print("  4. Calculate loads:")
    print("     (2,2): L = 2/3 ≈ 0.667")
    print("     (1,3): L = 3/3 = 1.0")
    print("     (3,1): L = 3/3 = 1.0")
    print("     (2,3): L = 3/3 = 1.0")
    print("     (3,2): L = 3/3 = 1.0")
    print("  5. (2,2) has minimum load while providing strong consistency.")
    print("  6. Also satisfies W > N/2 (2 > 1.5) to avoid write conflicts.")
    print("  7. Therefore, (2,2) is optimal. ✓")

if __name__ == "__main__":
    analyze_quorum_tradeoffs_comprehensive()
    simulate_dynamo_store()
    prove_quorum_properties()

4. Erasure Coding Theory: Complete Mathematical Foundation

The Mathematics of Reed-Solomon Codes

Erasure coding transforms k data blocks into n encoded blocks (n > k) such that any k out of n blocks can reconstruct the original data. This is based on polynomial interpolation over finite fields.

Mathematical Foundation:

Finite Field (Galois Field) GF(2^w):
- A field with 2^w elements
- Operations: addition (XOR), multiplication (log/antilog tables)
- Primitive polynomial defines the field
Vandermonde Matrix:
For encoding, we create matrix V where:

   V[i][j] = (i+1)^j in GF(2^w)

The encoding operation: encoded = V × data

Reconstruction: Any k rows of V form a k×k matrix V' We solve: data = V'⁻¹ × encoded'

Complete Python Implementation with Galois Field Arithmetic

import numpy as np
from typing import List, Optional, Tuple
import math
from dataclasses import dataclass
from functools import lru_cache
import random
import hashlib
import time

class GaloisField:
    """
    Complete implementation of GF(2^8) for Reed-Solomon coding
    Uses primitive polynomial x^8 + x^4 + x^3 + x^2 + 1
    """

    def __init__(self, w: int = 8):
        self.w = w
        self.size = 1 << w  # 2^w
        self.gflog = [0] * self.size
        self.gfilog = [0] * self.size
        self.primitive_poly = 0x11D  # x^8 + x^4 + x^3 + x^2 + 1

        self._init_tables()

    def _init_tables(self):
        """Initialize logarithm and inverse logarithm tables"""
        x = 1
        for i in range(self.size - 1):
            self.gflog[x] = i
            self.gfilog[i] = x

            x <<= 1
            if x & self.size:
                x ^= self.primitive_poly

        # Set log(0) to a large value (undefined)
        self.gflog[0] = 2 * self.size

    def add(self, a: int, b: int) -> int:
        """Addition in GF(2^w) is XOR"""
        return a ^ b

    def subtract(self, a: int, b: int) -> int:
        """Subtraction in GF(2^w) is XOR (same as addition)"""
        return a ^ b

    def multiply(self, a: int, b: int) -> int:
        """Multiplication in GF(2^w) using log/antilog tables"""
        if a == 0 or b == 0:
            return 0

        # log(a) + log(b) mod (2^w - 1)
        sum_log = (self.gflog[a] + self.gflog[b]) % (self.size - 1)

        return self.gfilog[sum_log]

    def divide(self, a: int, b: int) -> int:
        """Division in GF(2^w)"""
        if b == 0:
            raise ValueError("Division by zero")
        if a == 0:
            return 0

        # log(a) - log(b) mod (2^w - 1)
        diff_log = (self.gflog[a] - self.gflog[b]) % (self.size - 1)

        return self.gfilog[diff_log]

    def power(self, a: int, n: int) -> int:
        """Exponentiation in GF(2^w)"""
        if a == 0:
            return 0
        if n == 0:
            return 1

        # (log(a) * n) mod (2^w - 1)
        power_log = (self.gflog[a] * n) % (self.size - 1)

        return self.gfilog[power_log]

    def inverse(self, a: int) -> int:
        """Multiplicative inverse in GF(2^w)"""
        if a == 0:
            raise ValueError("Zero has no inverse")

        # -log(a) mod (2^w - 1)
        inv_log = (self.size - 1 - self.gflog[a]) % (self.size - 1)

        return self.gfilog[inv_log]

    def eval_polynomial(self, coeffs: List[int], x: int) -> int:
        """Evaluate polynomial at point x using Horner's method"""
        result = 0
        for coeff in reversed(coeffs):
            result = self.multiply(result, x)
            result = self.add(result, coeff)
        return result

    def interpolate(self, points: List[Tuple[int, int]]) -> List[int]:
        """
        Lagrange interpolation in GF(2^w)
        Returns polynomial coefficients
        """
        k = len(points)

        # Initialize result polynomial coefficients to 0
        coeffs = [0] * k

        for i in range(k):
            # Compute Lagrange basis polynomial L_i(x)
            xi, yi = points[i]

            # Compute numerator product (x - x_j) for j != i
            numerator = [1]  # Start with 1 (empty product)

            for j in range(k):
                if i == j:
                    continue
                xj = points[j][0]

                # Multiply numerator by (x - x_j)
                new_numerator = [0] * (len(numerator) + 1)
                for deg, coeff in enumerate(numerator):
                    # coeff * x
                    new_numerator[deg + 1] = self.add(new_numerator[deg + 1], coeff)
                    # coeff * (-xj)
                    term = self.multiply(coeff, xj)
                    new_numerator[deg] = self.add(new_numerator[deg], term)

                numerator = new_numerator

            # Compute denominator product (x_i - x_j) for j != i
            denominator = 1
            for j in range(k):
                if i == j:
                    continue
                xj = points[j][0]
                denominator = self.multiply(denominator, self.add(xi, xj))

            # Multiply numerator by y_i / denominator
            scale = self.divide(yi, denominator)

            # Add scaled numerator to result
            for deg in range(len(numerator)):
                coeffs[deg] = self.add(coeffs[deg], self.multiply(numerator[deg], scale))

        return coeffs

    def vandermonde_matrix(self, rows: int, cols: int) -> np.ndarray:
        """Generate Vandermonde matrix for encoding"""
        matrix = np.zeros((rows, cols), dtype=int)

        for i in range(rows):
            for j in range(cols):
                matrix[i, j] = self.power(i + 1, j)  # (i+1)^j

        return matrix

    def invert_matrix(self, matrix: np.ndarray) -> np.ndarray:
        """Invert matrix in GF(2^w) using Gaussian elimination"""
        n = matrix.shape[0]

        # Augment matrix with identity
        augmented = np.hstack([matrix, np.eye(n, dtype=int)])

        # Gaussian elimination
        for col in range(n):
            # Find pivot
            pivot = col
            while pivot < n and augmented[pivot, col] == 0:
                pivot += 1

            if pivot == n:
                raise ValueError("Matrix is singular")

            # Swap rows
            if pivot != col:
                augmented[[col, pivot]] = augmented[[pivot, col]]

            # Normalize pivot row
            pivot_val = augmented[col, col]
            inv_pivot = self.inverse(pivot_val)

            for j in range(2 * n):
                augmented[col, j] = self.multiply(augmented[col, j], inv_pivot)

            # Eliminate other rows
            for row in range(n):
                if row != col and augmented[row, col] != 0:
                    factor = augmented[row, col]
                    for j in range(2 * n):
                        product = self.multiply(factor, augmented[col, j])
                        augmented[row, j] = self.subtract(augmented[row, j], product)

        # Extract inverse
        inverse = augmented[:, n:]

        return inverse

class ReedSolomon:
    """
    Complete Reed-Solomon erasure coding implementation
    """

    def __init__(self, k: int, m: int):
        """
        Initialize Reed-Solomon encoder/decoder

        Args:
            k: Number of data shards
            m: Number of parity shards
        """
        self.k = k
        self.m = m
        self.n = k + m

        self.gf = GaloisField(8)

        # Generate encoding matrix
        self.encoding_matrix = self.gf.vandermonde_matrix(self.n, self.k)

        # Precompute decoding matrices for different erasure patterns
        self.decoding_cache = {}

    def encode(self, data_shards: List[bytes]) -> List[bytes]:
        """
        Encode data shards into parity shards

        Args:
            data_shards: List of k data shards (bytes)

        Returns:
            List of n encoded shards (data + parity)
        """
        if len(data_shards) != self.k:
            raise ValueError(f"Expected {self.k} data shards, got {len(data_shards)}")

        # Ensure all shards have same length
        shard_size = len(data_shards[0])
        for shard in data_shards:
            if len(shard) != shard_size:
                raise ValueError("All data shards must have the same length")

        # Create parity shards
        parity_shards = [bytearray(shard_size) for _ in range(self.m)]

        # Encode each byte position
        for byte_pos in range(shard_size):
            # Prepare vector of bytes at this position
            data_vector = [shard[byte_pos] for shard in data_shards]

            # Compute parity: parity = encoding_matrix[k:] × data
            for i in range(self.m):
                parity = 0
                for j in range(self.k):
                    product = self.gf.multiply(self.encoding_matrix[self.k + i, j], 
                                             data_vector[j])
                    parity = self.gf.add(parity, product)
                parity_shards[i][byte_pos] = parity

        # Return all shards
        return data_shards + parity_shards

    def decode(self, shards: List[Optional[bytes]], shard_size: int) -> List[bytes]:
        """
        Decode original data from available shards

        Args:
            shards: List of n shards (some may be None for missing)
            shard_size: Size of each shard

        Returns:
            List of k recovered data shards
        """
        # Identify which shards are available
        available = [i for i, shard in enumerate(shards) if shard is not None]

        if len(available) < self.k:
            raise ValueError(f"Need at least {self.k} shards, have {len(available)}")

        # If we have exactly k shards and they're all data shards
        if (len(available) == self.k and 
            all(i < self.k for i in available)):
            # No decoding needed, just return data shards
            return [shards[i] for i in range(self.k)]

        # Select first k available shards
        selected = available[:self.k]

        # Create submatrix of encoding matrix for selected shards
        submatrix = self.encoding_matrix[selected, :self.k]

        # Get inverse of submatrix
        cache_key = tuple(sorted(selected))
        if cache_key in self.decoding_cache:
            inverse = self.decoding_cache[cache_key]
        else:
            inverse = self.gf.invert_matrix(submatrix)
            self.decoding_cache[cache_key] = inverse

        # Recover data shards
        data_shards = [bytearray(shard_size) for _ in range(self.k)]

        for byte_pos in range(shard_size):
            # Get bytes from selected shards
            selected_bytes = [shards[i][byte_pos] for i in selected]

            # Recover data: data = inverse × selected_bytes
            for i in range(self.k):
                recovered = 0
                for j in range(self.k):
                    product = self.gf.multiply(inverse[i, j], selected_bytes[j])
                    recovered = self.gf.add(recovered, product)
                data_shards[i][byte_pos] = recovered

        return [bytes(shard) for shard in data_shards]

    def repair(self, shards: List[Optional[bytes]], shard_size: int) -> List[bytes]:
        """
        Repair all missing shards (both data and parity)

        Returns:
            Complete list of n repaired shards
        """
        # First decode data
        data_shards = self.decode(shards, shard_size)

        # Then re-encode to get all shards
        return self.encode(data_shards)

    def analyze(self, node_failure_prob: float = 0.01) -> dict:
        """
        Analyze reliability and storage efficiency
        """
        # Storage overhead
        overhead = self.n / self.k

        # Probability system survives f failures
        survival_probs = []
        for f in range(self.m + 1):
            # Probability exactly f failures (binomial distribution)
            prob_f = (math.comb(self.n, f) * 
                     (node_failure_prob ** f) * 
                     ((1 - node_failure_prob) ** (self.n - f)))

            # System survives if f ≤ m
            if f <= self.m:
                survival_probs.append(prob_f)

        survival_prob = sum(survival_probs)

        # Annual durability (assuming independent failures)
        annual_durability = 1 - (1 - survival_prob) ** 365

        # Mean time to data loss (MTTDL)
        # Simplified model: MTTDL ≈ 1 / (prob_of_data_loss_per_year)
        prob_data_loss_per_year = 1 - annual_durability
        if prob_data_loss_per_year > 0:
            mttdl_years = 1 / prob_data_loss_per_year
        else:
            mttdl_years = float('inf')

        return {
            'data_shards': self.k,
            'parity_shards': self.m,
            'total_shards': self.n,
            'storage_overhead': overhead,
            'survivable_failures': self.m,
            'survival_probability': survival_prob,
            'annual_durability': annual_durability,
            'mttdl_years': mttdl_years,
            'efficiency': self.k / self.n
        }

class DistributedStorageSystem:
    """
    Complete distributed storage system with erasure coding
    """

    def __init__(self, k: int = 6, m: int = 3, node_count: int = 12):
        """
        Initialize distributed storage

        Args:
            k: Data shards
            m: Parity shards
            node_count: Total storage nodes
        """
        self.k = k
        self.m = m
        self.n = k + m
        self.node_count = node_count

        self.rs = ReedSolomon(k, m)
        self.nodes = [{} for _ in range(node_count)]
        self.node_status = [True] * node_count
        self.node_capacity = [10 * 1024 * 1024] * node_count  # 10MB each
        self.node_used = [0] * node_count

        # Statistics
        self.stats = {
            'files_stored': 0,
            'bytes_stored': 0,
            'shards_stored': 0,
            'repair_operations': 0,
            'failed_repairs': 0,
            'node_failures': 0
        }

    def _hash_shard_location(self, file_id: str, shard_index: int) -> int:
        """Consistent hashing for shard placement"""
        # Create unique identifier for this shard
        key = f"{file_id}:{shard_index}"

        # Use MD5 hash for even distribution
        hash_bytes = hashlib.md5(key.encode()).digest()
        hash_int = int.from_bytes(hash_bytes, 'big')

        return hash_int % self.node_count

    def _find_placement(self, file_id: str, shard_index: int, 
                       used_nodes: set) -> Optional[int]:
        """
        Find node for shard placement using consistent hashing
        with fallback for failed nodes
        """
        # Try primary location
        primary = self._hash_shard_location(file_id, shard_index)

        # Check primary node
        if (self.node_status[primary] and 
            primary not in used_nodes and
            self.node_used[primary] < self.node_capacity[primary]):
            return primary

        # Primary not available, try alternative locations
        for offset in range(1, self.node_count):
            node_idx = (primary + offset) % self.node_count

            if (self.node_status[node_idx] and 
                node_idx not in used_nodes and
                self.node_used[node_idx] < self.node_capacity[node_idx]):
                return node_idx

        return None

    def store_file(self, file_id: str, data: bytes) -> bool:
        """
        Store file with erasure coding across nodes
        """
        # Split data into k shards
        shard_size = math.ceil(len(data) / self.k)
        padded_data = data.ljust(shard_size * self.k, b'\x00')

        data_shards = []
        for i in range(self.k):
            start = i * shard_size
            end = start + shard_size
            data_shards.append(padded_data[start:end])

        # Encode to get all shards
        all_shards = self.rs.encode(data_shards)

        # Place shards on nodes
        placements = []
        used_nodes = set()

        for i, shard in enumerate(all_shards):
            node_idx = self._find_placement(file_id, i, used_nodes)

            if node_idx is None:
                # Could not find suitable node
                return False

            # Store shard
            self.nodes[node_idx][file_id] = {
                'shard': shard,
                'index': i,
                'size': len(shard),
                'is_data': i < self.k
            }

            self.node_used[node_idx] += len(shard)
            used_nodes.add(node_idx)
            placements.append(node_idx)

        # Store metadata
        metadata = {
            'file_id': file_id,
            'size': len(data),
            'shard_size': shard_size,
            'placements': placements,
            'timestamp': time.time()
        }

        # Store metadata on multiple nodes
        metadata_nodes = placements[:3]  # Store on first 3 nodes
        for node_idx in metadata_nodes:
            if 'metadata' not in self.nodes[node_idx]:
                self.nodes[node_idx]['metadata'] = {}
            self.nodes[node_idx]['metadata'][file_id] = metadata

        # Update statistics
        self.stats['files_stored'] += 1
        self.stats['bytes_stored'] += len(data)
        self.stats['shards_stored'] += len(all_shards)

        return True

    def retrieve_file(self, file_id: str) -> Optional[bytes]:
        """
        Retrieve file, tolerating node failures
        """
        # First, find metadata
        metadata = None
        for node_idx in range(self.node_count):
            if (self.node_status[node_idx] and 
                'metadata' in self.nodes[node_idx] and
                file_id in self.nodes[node_idx]['metadata']):
                metadata = self.nodes[node_idx]['metadata'][file_id]
                break

        if metadata is None:
            return None

        # Collect available shards
        shards = [None] * self.n
        shard_size = metadata['shard_size']

        for node_idx in metadata['placements']:
            if (self.node_status[node_idx] and 
                file_id in self.nodes[node_idx]):
                shard_data = self.nodes[node_idx][file_id]
                shards[shard_data['index']] = shard_data['shard']

        # Check if we have enough shards
        available = sum(1 for s in shards if s is not None)

        if available < self.k:
            # Not enough shards, try to repair first
            self._trigger_repair(file_id, metadata)
            return None

        # Decode data
        try:
            data_shards = self.rs.decode(shards, shard_size)

            # Combine data shards
            data = bytearray()
            for shard in data_shards:
                data.extend(shard)

            # Remove padding
            data = data[:metadata['size']]

            return bytes(data)

        except Exception as e:
            print(f"Decode failed: {e}")
            return None

    def _trigger_repair(self, file_id: str, metadata: dict):
        """
        Trigger repair of missing shards
        """
        self.stats['repair_operations'] += 1

        # Collect available shards
        shards = [None] * self.n
        shard_size = metadata['shard_size']

        for node_idx in metadata['placements']:
            if (self.node_status[node_idx] and 
                file_id in self.nodes[node_idx]):
                shard_data = self.nodes[node_idx][file_id]
                shards[shard_data['index']] = shard_data['shard']

        # Check if we have enough to repair
        available = sum(1 for s in shards if s is not None)

        if available < self.k:
            self.stats['failed_repairs'] += 1
            return False

        # Repair all shards
        try:
            repaired_shards = self.rs.repair(shards, shard_size)

            # Store repaired shards on new nodes if needed
            for i, shard in enumerate(repaired_shards):
                if shards[i] is None:
                    # This shard was missing, find new placement
                    node_idx = self._find_placement(file_id, i, set())

                    if node_idx is not None:
                        self.nodes[node_idx][file_id] = {
                            'shard': shard,
                            'index': i,
                            'size': len(shard),
                            'is_data': i < self.k
                        }
                        self.node_used[node_idx] += len(shard)

                        # Update placement in metadata
                        metadata['placements'][i] = node_idx

            # Update metadata on nodes
            metadata_nodes = metadata['placements'][:3]
            for node_idx in metadata_nodes:
                if 'metadata' not in self.nodes[node_idx]:
                    self.nodes[node_idx]['metadata'] = {}
                self.nodes[node_idx]['metadata'][file_id] = metadata

            return True

        except Exception as e:
            print(f"Repair failed: {e}")
            self.stats['failed_repairs'] += 1
            return False

    def simulate_node_failure(self, node_indices: List[int]):
        """
        Simulate node failures
        """
        for idx in node_indices:
            self.node_status[idx] = False
            self.stats['node_failures'] += 1

            # Clear node data
            self.nodes[idx] = {}
            self.node_used[idx] = 0

    def simulate_node_recovery(self, node_indices: List[int]):
        """
        Simulate node recovery
        """
        for idx in node_indices:
            self.node_status[idx] = True

    def get_statistics(self) -> dict:
        """
        Get system statistics
        """
        # Calculate storage efficiency
        total_capacity = sum(self.node_capacity)
        total_used = sum(self.node_used)

        # Calculate data durability estimate
        analysis = self.rs.analyze(node_failure_prob=0.01)

        return {
            **self.stats,
            'total_capacity': total_capacity,
            'total_used': total_used,
            'utilization': total_used / total_capacity,
            'expected_durability': analysis['annual_durability'],
            'storage_overhead': analysis['storage_overhead'],
            'survivable_failures': analysis['survivable_failures']
        }

    def get_system_health(self) -> dict:
        """
        Get comprehensive system health report
        """
        # Count healthy nodes
        healthy_nodes = sum(self.node_status)

        # Check files for missing shards
        files_at_risk = 0
        files_unrecoverable = 0

        # For each file, check shard availability
        all_files = set()
        for node in self.nodes:
            if 'metadata' in node:
                all_files.update(node['metadata'].keys())

        for file_id in all_files:
            # Find metadata
            metadata = None
            for node_idx in range(self.node_count):
                if (self.node_status[node_idx] and 
                    'metadata' in self.nodes[node_idx] and
                    file_id in self.nodes[node_idx]['metadata']):
                    metadata = self.nodes[node_idx]['metadata'][file_id]
                    break

            if metadata is None:
                continue

            # Count available shards
            available = 0
            for node_idx in metadata['placements']:
                if (self.node_status[node_idx] and 
                    file_id in self.nodes[node_idx]):
                    available += 1

            if available < self.k:
                files_unrecoverable += 1
            elif available < self.n:
                files_at_risk += 1

        return {
            'healthy_nodes': healthy_nodes,
            'failed_nodes': self.node_count - healthy_nodes,
            'files_at_risk': files_at_risk,
            'files_unrecoverable': files_unrecoverable,
            'total_files': len(all_files),
            'node_failure_rate': self.stats['node_failures'] / max(1, self.stats['files_stored'])
        }

def compare_storage_strategies():
    """
    Compare replication vs erasure coding
    """
    print("="*70)
    print("STORAGE STRATEGY COMPARISON: REPLICATION VS ERASURE CODING")
    print("="*70)

    # Test configurations
    data_size = 1024 * 1024  # 1MB
    test_data = b"X" * data_size

    strategies = [
        {
            'name': '3x Replication',
            'type': 'replication',
            'copies': 3,
            'storage': data_size * 3
        },
        {
            'name': 'EC (4+2)',
            'type': 'erasure',
            'k': 4,
            'm': 2,
            'storage': data_size * 6 / 4
        },
        {
            'name': 'EC (6+3)',
            'type': 'erasure',
            'k': 6,
            'm': 3,
            'storage': data_size * 9 / 6
        },
        {
            'name': 'EC (10+4)',
            'type': 'erasure',
            'k': 10,
            'm': 4,
            'storage': data_size * 14 / 10
        }
    ]

    # Node failure probability
    p_fail = 0.01

    print("\nComparison Table:")
    print("-"*120)
    print(f"{'Strategy':<15} {'Storage':<12} {'Overhead':<10} {'Survives':<10} "
          f"{'Annual Durability':<18} {'MTTDL (years)':<15} {'Efficiency':<10}")
    print("-"*120)

    for strategy in strategies:
        if strategy['type'] == 'replication':
            copies = strategy['copies']
            overhead = copies

            # Probability all copies fail
            all_fail = p_fail ** copies
            survival = 1 - all_fail

            annual_durability = 1 - (1 - survival) ** 365

            if all_fail > 0:
                mttdl = 1 / (all_fail * 365)  # Approximate
            else:
                mttdl = float('inf')

            efficiency = 1 / copies

            print(f"{strategy['name']:<15} {strategy['storage']/1024/1024:<11.2f}MB "
                  f"{overhead:<10.2f}x {copies-1:<10} "
                  f"{annual_durability:<18.6f} {mttdl:<15.2e} {efficiency:<10.2f}")

        else:
            k = strategy['k']
            m = strategy['m']
            n = k + m

            rs = ReedSolomon(k, m)
            analysis = rs.analyze(p_fail)

            print(f"{strategy['name']:<15} {strategy['storage']/1024/1024:<11.2f}MB "
                  f"{analysis['storage_overhead']:<10.2f}x {m:<10} "
                  f"{analysis['annual_durability']:<18.6f} "
                  f"{analysis['mttdl_years']:<15.2e} "
                  f"{analysis['efficiency']:<10.2f}")

    # Visual comparison
    visualize_comparison(strategies, p_fail)

def visualize_comparison(strategies, p_fail):
    """Create visualization of storage strategies"""
    import matplotlib.pyplot as plt

    fig, axes = plt.subplots(2, 2, figsize=(12, 10))

    # Prepare data
    names = [s['name'] for s in strategies]

    # Calculate metrics for each strategy
    storages = []
    durabilities = []
    efficiencies = []
    costs = []  # Cost = storage * (1 - durability)

    for strategy in strategies:
        if strategy['type'] == 'replication':
            copies = strategy['copies']
            all_fail = p_fail ** copies
            survival = 1 - all_fail
            annual_durability = 1 - (1 - survival) ** 365
            efficiency = 1 / copies

            storages.append(strategy['storage'] / 1024 / 1024)  # MB
            durabilities.append(annual_durability)
            efficiencies.append(efficiency)
            costs.append(strategy['storage'] * (1 - annual_durability))

        else:
            k = strategy['k']
            m = strategy['m']

            rs = ReedSolomon(k, m)
            analysis = rs.analyze(p_fail)

            storages.append(strategy['storage'] / 1024 / 1024)
            durabilities.append(analysis['annual_durability'])
            efficiencies.append(analysis['efficiency'])
            costs.append(strategy['storage'] * (1 - analysis['annual_durability']))

    # Plot 1: Storage vs Durability
    ax1 = axes[0, 0]
    scatter = ax1.scatter(storages, durabilities, s=100, alpha=0.6)
    ax1.set_xlabel('Storage Required (MB)')
    ax1.set_ylabel('Annual Durability')
    ax1.set_title('Storage vs Durability Tradeoff')
    ax1.grid(True, alpha=0.3)

    # Annotate points
    for i, name in enumerate(names):
        ax1.annotate(name, (storages[i], durabilities[i]), 
                    xytext=(5, 5), textcoords='offset points')

    # Plot 2: Efficiency vs Durability
    ax2 = axes[0, 1]
    ax2.scatter(efficiencies, durabilities, s=100, alpha=0.6)
    ax2.set_xlabel('Storage Efficiency')
    ax2.set_ylabel('Annual Durability')
    ax2.set_title('Efficiency vs Durability')
    ax2.grid(True, alpha=0.3)

    for i, name in enumerate(names):
        ax2.annotate(name, (efficiencies[i], durabilities[i]), 
                    xytext=(5, 5), textcoords='offset points')

    # Plot 3: Cost Analysis
    ax3 = axes[1, 0]
    bars = ax3.bar(names, [c / 1024 / 1024 for c in costs])
    ax3.set_xlabel('Strategy')
    ax3.set_ylabel('Expected Loss (MB)')
    ax3.set_title('Expected Data Loss Cost')
    ax3.tick_params(axis='x', rotation=45)
    ax3.grid(True, alpha=0.3, axis='y')

    # Add value labels on bars
    for bar in bars:
        height = bar.get_height()
        ax3.text(bar.get_x() + bar.get_width()/2., height,
                f'{height:.2f}', ha='center', va='bottom')

    # Plot 4: Survival probability vs failures
    ax4 = axes[1, 1]

    # Simulate different numbers of failures
    max_failures = 10
    failure_counts = list(range(max_failures + 1))

    for strategy in strategies[:2]:  # Just show first two for clarity
        survival_probs = []

        for f in failure_counts:
            if strategy['type'] == 'replication':
                # Replication survives if at least 1 copy survives
                copies = strategy['copies']
                prob_survive = 1 - (p_fail ** min(f, copies))
                survival_probs.append(prob_survive)

            else:
                # Erasure coding survives if f ≤ m
                k = strategy['k']
                m = strategy['m']
                if f <= m:
                    survival_probs.append(1.0)
                else:
                    survival_probs.append(0.0)

        ax4.plot(failure_counts, survival_probs, 
                label=strategy['name'], linewidth=2)

    ax4.set_xlabel('Number of Node Failures')
    ax4.set_ylabel('Survival Probability')
    ax4.set_title('Failure Tolerance')
    ax4.legend()
    ax4.grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()

def run_distributed_storage_simulation():
    """
    Simulate complete distributed storage system
    """
    print("\n" + "="*70)
    print("DISTRIBUTED STORAGE SIMULATION")
    print("="*70)

    # Create storage system
    storage = DistributedStorageSystem(k=4, m=2, node_count=10)

    print("\n1. Storing files...")

    # Store some files
    files = [
        ("doc1.txt", b"This is document 1 with important content."),
        ("img1.jpg", b"FAKE_IMAGE_DATA" * 100),  # 1.6KB
        ("data.bin", b"\x00\x01\x02\x03" * 500),  # 2KB
    ]

    for file_id, data in files:
        success = storage.store_file(file_id, data)
        print(f"  Stored {file_id} ({len(data)} bytes): {'✓' if success else '✗'}")

    print("\n2. Retrieving files normally...")
    for file_id, _ in files:
        data = storage.retrieve_file(file_id)
        if data:
            print(f"  Retrieved {file_id}: {len(data)} bytes")
        else:
            print(f"  Failed to retrieve {file_id}")

    print("\n3. Simulating node failures...")
    # Fail 3 random nodes
    failed_nodes = random.sample(range(10), 3)
    storage.simulate_node_failure(failed_nodes)
    print(f"  Failed nodes: {failed_nodes}")

    print("\n4. Retrieving after failures...")
    for file_id, _ in files:
        data = storage.retrieve_file(file_id)
        if data:
            print(f"  Retrieved {file_id} after failures: ✓")
        else:
            print(f"  Failed to retrieve {file_id}: ✗")

    print("\n5. System health check...")
    health = storage.get_system_health()
    for key, value in health.items():
        print(f"  {key}: {value}")

    print("\n6. Triggering repair...")
    # Try to repair missing shards
    for file_id, _ in files:
        # Find metadata to trigger repair
        for node_idx in range(10):
            if (storage.node_status[node_idx] and 
                'metadata' in storage.nodes[node_idx] and
                file_id in storage.nodes[node_idx]['metadata']):
                metadata = storage.nodes[node_idx]['metadata'][file_id]
                storage._trigger_repair(file_id, metadata)
                break

    print("\n7. Statistics:")
    stats = storage.get_statistics()
    for key, value in stats.items():
        print(f"  {key}: {value}")

def demonstrate_reed_solomon_math():
    """
    Demonstrate Reed-Solomon mathematics step by step
    """
    print("\n" + "="*70)
    print("REED-SOLOMON MATHEMATICS DEMONSTRATION")
    print("="*70)

    # Create Galois Field
    gf = GaloisField(8)

    print("\n1. Galois Field GF(2^8) operations:")
    a, b = 5, 10
    print(f"   {a} + {b} = {gf.add(a, b)} (XOR)")
    print(f"   {a} × {b} = {gf.multiply(a, b)}")
    print(f"   {a}⁻¹ = {gf.inverse(a)}")
    print(f"   Verify: {a} × {a}⁻¹ = {gf.multiply(a, gf.inverse(a))} (should be 1)")

    print("\n2. Polynomial evaluation:")
    coeffs = [1, 2, 3]  # Represents 1 + 2x + 3x²
    x = 4
    result = gf.eval_polynomial(coeffs, x)
    print(f"   P(x) = 1 + 2x + 3x²")
    print(f"   P({x}) = {result}")

    print("\n3. Lagrange interpolation:")
    points = [(1, 2), (2, 3), (3, 5)]  # (x, y) points
    coeffs = gf.interpolate(points)
    print(f"   Points: {points}")
    print(f"   Interpolated polynomial coefficients: {coeffs}")

    # Verify
    print("\n   Verification:")
    for x, y in points:
        computed = gf.eval_polynomial(coeffs, x)
        print(f"   P({x}) = {computed} (expected {y}) {'✓' if computed == y else '✗'}")

    print("\n4. Vandermonde matrix for (4,2) Reed-Solomon:")
    rs = ReedSolomon(4, 2)
    print("   Encoding matrix (6×4):")
    print(rs.encoding_matrix)

    print("\n5. Encoding example:")
    data = [1, 2, 3, 4]  # Simple data
    print(f"   Data: {data}")

    # Encode
    encoded = []
    for i in range(6):
        row_sum = 0
        for j in range(4):
            row_sum = gf.add(row_sum, gf.multiply(rs.encoding_matrix[i, j], data[j]))
        encoded.append(row_sum)

    print(f"   Encoded: {encoded}")

    print("\n6. Decoding with erasures:")
    # Simulate losing shards 0 and 2
    received = encoded.copy()
    received[0] = None
    received[2] = None
    print(f"   Received (with erasures): {received}")

    # Use remaining shards 1, 3, 4, 5
    available = [1, 3, 4, 5]
    submatrix = rs.encoding_matrix[available, :4]
    print(f"   Submatrix from available shards:")
    print(submatrix)

    # Invert submatrix
    inverse = gf.invert_matrix(submatrix)
    print(f"   Inverse of submatrix:")
    print(inverse)

    # Recover data
    received_data = [received[i] for i in available]
    recovered = [0, 0, 0, 0]

    for i in range(4):
        for j in range(4):
            recovered[i] = gf.add(recovered[i], 
                                gf.multiply(inverse[i, j], received_data[j]))

    print(f"   Recovered data: {recovered}")
    print(f"   Original data:  {data}")
    print(f"   Match: {'✓' if recovered == data else '✗'}")

if __name__ == "__main__":
    compare_storage_strategies()
    run_distributed_storage_simulation()
    demonstrate_reed_solomon_math()

5. Byzantine Agreement: Complete Implementation with Proofs

The Byzantine Generals Problem: Mathematical Foundation

Problem Statement: N generals, some of which may be traitors (Byzantine), need to agree on a common plan of action.

Impossibility Results:

Theorem (Lamport 1982): With 3 generals and 1 traitor, no solution exists.
Theorem: Solution exists iff N ≥ 3f + 1, where f is number of traitors.

Why 3f + 1?

f traitors could lie
f non-responding (slow/crashed)
Need f+1 honest, responding generals to outnumber traitors

Complete C++ Implementation: Practical Byzantine Fault Tolerance (PBFT)

#include <iostream>
#include <vector>
#include <map>
#include <set>
#include <string>
#include <memory>
#include <random>
#include <chrono>
#include <algorithm>
#include <openssl/sha.h>
#include <openssl/evp.h>
#include <iomanip>
#include <queue>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <atomic>

// ============================================================================
// CRYPTOGRAPHIC PRIMITIVES
// ============================================================================

class CryptographicHash {
public:
    static std::string sha256(const std::string& data) {
        unsigned char hash[SHA256_DIGEST_LENGTH];
        SHA256_CTX sha256;

        SHA256_Init(&sha256);
        SHA256_Update(&sha256, data.c_str(), data.size());
        SHA256_Final(hash, &sha256);

        std::stringstream ss;
        for(int i = 0; i < SHA256_DIGEST_LENGTH; i++) {
            ss << std::hex << std::setw(2) << std::setfill('0') 
               << static_cast<int>(hash[i]);
        }

        return ss.str();
    }

    static std::string sign(const std::string& data, int node_id) {
        // Simplified signing: hash(data + secret)
        std::string secret = "node_secret_" + std::to_string(node_id);
        return sha256(data + secret);
    }

    static bool verify(const std::string& data, 
                      const std::string& signature, 
                      int node_id) {
        std::string computed = sign(data, node_id);
        return computed == signature;
    }
};

// ============================================================================
// MESSAGE TYPES AND STRUCTURES
// ============================================================================

struct Message {
    enum Type {
        REQUEST,        // Client request
        PREPREPARE,     // Leader proposes
        PREPARE,        // Replicas prepare
        COMMIT,         // Replicas commit
        REPLY,          // Reply to client
        VIEW_CHANGE,    // View change protocol
        NEW_VIEW        // New view establishment
    };

    Type type;
    int view_number;
    int sequence_number;
    std::string digest;
    std::string content;
    int sender_id;
    std::string signature;
    std::string client_id;
    uint64_t timestamp;

    Message() = default;

    Message(Type t, int view, int seq, const std::string& d, 
            const std::string& c, int sender)
        : type(t), view_number(view), sequence_number(seq),
          digest(d), content(c), sender_id(sender),
          timestamp(std::chrono::system_clock::now().time_since_epoch().count()) {}

    std::string hash() const {
        std::string data = 
            std::to_string(static_cast<int>(type)) +
            std::to_string(view_number) +
            std::to_string(sequence_number) +
            digest +
            content +
            std::to_string(sender_id) +
            std::to_string(timestamp);

        return CryptographicHash::sha256(data);
    }

    void sign_message(int node_id) {
        std::string data = hash();
        signature = CryptographicHash::sign(data, node_id);
    }

    bool verify_signature() const {
        std::string data = hash();
        return CryptographicHash::verify(data, signature, sender_id);
    }

    std::string to_string() const {
        std::string type_str;
        switch(type) {
            case REQUEST: type_str = "REQUEST"; break;
            case PREPREPARE: type_str = "PREPREPARE"; break;
            case PREPARE: type_str = "PREPARE"; break;
            case COMMIT: type_str = "COMMIT"; break;
            case REPLY: type_str = "REPLY"; break;
            case VIEW_CHANGE: type_str = "VIEW_CHANGE"; break;
            case NEW_VIEW: type_str = "NEW_VIEW"; break;
        }

        return "Message{" + type_str + 
               ", view=" + std::to_string(view_number) +
               ", seq=" + std::to_string(sequence_number) +
               ", digest=" + digest.substr(0, 8) + "..." +
               ", sender=" + std::to_string(sender_id) + "}";
    }
};

// ============================================================================
// PBFT STATE MACHINE
// ============================================================================

class PBFTState {
private:
    int node_id;
    int total_nodes;
    int max_faulty;

    // Current state
    int current_view;
    int last_executed;
    int last_stable_checkpoint;

    // Message logs
    std::map<int, std::map<std::string, std::set<int>>> prepare_certificates;
    std::map<int, std::map<std::string, std::set<int>>> commit_certificates;

    // Checkpoints
    std::map<int, std::string> checkpoints;  // sequence -> state_hash

    // Client requests
    std::map<std::string, std::pair<int, Message>> client_requests;

    // Pending requests
    std::map<int, Message> pending_requests;

    // View change state
    bool view_change_pending;
    std::map<int, std::set<int>> view_change_certificates;

    // Watermarks
    int low_watermark;
    int high_watermark;

    std::mutex state_mutex;

public:
    PBFTState(int id, int n)
        : node_id(id), total_nodes(n), max_faulty((n - 1) / 3),
          current_view(0), last_executed(0), last_stable_checkpoint(0),
          view_change_pending(false), low_watermark(0), high_watermark(100) {

        if (n <= 3 * max_faulty) {
            throw std::runtime_error("N must be > 3f for Byzantine tolerance");
        }
    }

    bool is_primary() const {
        return current_view % total_nodes == node_id;
    }

    int primary_of_view(int view) const {
        return view % total_nodes;
    }

    bool can_prepare(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        // Check view number
        if (msg.view_number != current_view) {
            return false;
        }

        // Check sequence number is within watermarks
        if (msg.sequence_number <= low_watermark || 
            msg.sequence_number > high_watermark) {
            return false;
        }

        // Check if we already prepared for this sequence
        auto it = prepare_certificates.find(msg.sequence_number);
        if (it != prepare_certificates.end()) {
            auto& certs = it->second;
            if (certs.find(msg.digest) != certs.end()) {
                // Already prepared this digest
                return false;
            }
        }

        return true;
    }

    void add_prepare(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        prepare_certificates[msg.sequence_number][msg.digest].insert(msg.sender_id);
    }

    bool has_prepare_certificate(int sequence, const std::string& digest) {
        std::lock_guard<std::mutex> lock(state_mutex);

        auto seq_it = prepare_certificates.find(sequence);
        if (seq_it == prepare_certificates.end()) {
            return false;
        }

        auto& certs = seq_it->second;
        auto cert_it = certs.find(digest);
        if (cert_it == certs.end()) {
            return false;
        }

        // Need 2f matching prepares (excluding our own)
        int count = cert_it->second.size();
        if (cert_it->second.find(node_id) != cert_it->second.end()) {
            count--;  // Don't count our own
        }

        return count >= 2 * max_faulty;
    }

    void add_commit(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        commit_certificates[msg.sequence_number][msg.digest].insert(msg.sender_id);
    }

    bool has_commit_certificate(int sequence, const std::string& digest) {
        std::lock_guard<std::mutex> lock(state_mutex);

        auto seq_it = commit_certificates.find(sequence);
        if (seq_it == commit_certificates.end()) {
            return false;
        }

        auto& certs = seq_it->second;
        auto cert_it = certs.find(digest);
        if (cert_it == certs.end()) {
            return false;
        }

        // Need 2f+1 matching commits
        return cert_it->second.size() >= 2 * max_faulty + 1;
    }

    bool can_execute(int sequence) {
        std::lock_guard<std::mutex> lock(state_mutex);
        return sequence == last_executed + 1;
    }

    void mark_executed(int sequence) {
        std::lock_guard<std::mutex> lock(state_mutex);
        if (sequence == last_executed + 1) {
            last_executed = sequence;

            // Create checkpoint every 100 requests
            if (sequence % 100 == 0) {
                create_checkpoint(sequence);
            }
        }
    }

    void create_checkpoint(int sequence) {
        // In real implementation, this would save application state
        std::string state_hash = CryptographicHash::sha256(
            "checkpoint_at_" + std::to_string(sequence));

        checkpoints[sequence] = state_hash;
        last_stable_checkpoint = sequence;

        // Move watermarks
        low_watermark = last_stable_checkpoint;
        high_watermark = low_watermark + 100;
    }

    void start_view_change(int new_view) {
        std::lock_guard<std::mutex> lock(state_mutex);
        view_change_pending = true;
        current_view = new_view;
    }

    void add_view_change(int view, int node) {
        std::lock_guard<std::mutex> lock(state_mutex);
        view_change_certificates[view].insert(node);
    }

    bool has_view_change_certificate(int view) {
        std::lock_guard<std::mutex> lock(state_mutex);
        auto it = view_change_certificates.find(view);
        if (it == view_change_certificates.end()) {
            return false;
        }
        return it->second.size() >= 2 * max_faulty + 1;
    }

    int get_current_view() const { return current_view; }
    int get_last_executed() const { return last_executed; }
    int get_low_watermark() const { return low_watermark; }
    int get_high_watermark() const { return high_watermark; }
    int get_total_nodes() const { return total_nodes; }
    int get_max_faulty() const { return max_faulty; }
};

// ============================================================================
// PBFT NODE IMPLEMENTATION
// ============================================================================

class PBFTNode {
private:
    int node_id;
    bool byzantine;
    PBFTState state;

    // Network
    std::function<void(const Message&)> broadcast_callback;
    std::function<void(int, const Message&)> send_callback;

    // Request queue
    std::queue<Message> request_queue;
    std::mutex queue_mutex;
    std::condition_variable queue_cv;

    // Worker thread
    std::thread worker_thread;
    std::atomic<bool> running;

    // Byzantine behavior
    std::mt19937 rng;
    std::uniform_real_distribution<double> dist;

    // Statistics
    struct Stats {
        int messages_received = 0;
        int messages_sent = 0;
        int requests_executed = 0;
        int byzantine_actions = 0;
        int view_changes = 0;
    } stats;

public:
    PBFTNode(int id, int total_nodes, bool is_byzantine = false)
        : node_id(id), byzantine(is_byzantine), 
          state(id, total_nodes), running(false),
          rng(std::chrono::system_clock::now().time_since_epoch().count()),
          dist(0.0, 1.0) {}

    ~PBFTNode() {
        stop();
    }

    void set_network_callbacks(
        std::function<void(const Message&)> broadcast,
        std::function<void(int, const Message&)> send) {

        broadcast_callback = broadcast;
        send_callback = send;
    }

    void start() {
        running = true;
        worker_thread = std::thread(&PBFTNode::process_loop, this);
    }

    void stop() {
        running = false;
        queue_cv.notify_all();
        if (worker_thread.joinable()) {
            worker_thread.join();
        }
    }

    void receive_message(const Message& msg) {
        if (!msg.verify_signature()) {
            std::cerr << "Node " << node_id << ": Invalid signature from " 
                      << msg.sender_id << std::endl;
            return;
        }

        stats.messages_received++;

        // Byzantine nodes might drop messages
        if (byzantine && dist(rng) < 0.2) {  // 20% chance to drop
            stats.byzantine_actions++;
            return;
        }

        std::lock_guard<std::mutex> lock(queue_mutex);
        request_queue.push(msg);
        queue_cv.notify_one();
    }

    void process_loop() {
        while (running) {
            Message msg;
            {
                std::unique_lock<std::mutex> lock(queue_mutex);
                queue_cv.wait(lock, [this]() { 
                    return !request_queue.empty() || !running; 
                });

                if (!running) break;

                msg = request_queue.front();
                request_queue.pop();
            }

            process_message(msg);
        }
    }

    void process_message(const Message& msg) {
        switch (msg.type) {
            case Message::REQUEST:
                handle_request(msg);
                break;
            case Message::PREPREPARE:
                handle_preprepare(msg);
                break;
            case Message::PREPARE:
                handle_prepare(msg);
                break;
            case Message::COMMIT:
                handle_commit(msg);
                break;
            case Message::VIEW_CHANGE:
                handle_view_change(msg);
                break;
            case Message::NEW_VIEW:
                handle_new_view(msg);
                break;
            default:
                std::cerr << "Unknown message type: " << msg.type << std::endl;
        }
    }

    void handle_request(const Message& msg) {
        if (!state.is_primary()) {
            // Forward to primary
            int primary = state.primary_of_view(state.get_current_view());
            if (send_callback) {
                send_callback(primary, msg);
            }
            return;
        }

        // Primary assigns sequence number
        int sequence = state.get_last_executed() + 1;

        // Create PRE-PREPARE message
        Message preprepare(Message::PREPREPARE,
                          state.get_current_view(),
                          sequence,
                          msg.digest,
                          msg.content,
                          node_id);

        preprepare.sign_message(node_id);

        // Byzantine primary might assign wrong sequence
        if (byzantine && dist(rng) < 0.3) {
            stats.byzantine_actions++;
            preprepare.sequence_number = sequence + 100;  // Wrong sequence
        }

        if (broadcast_callback) {
            broadcast_callback(preprepare);
            stats.messages_sent++;
        }
    }

    void handle_preprepare(const Message& msg) {
        // Verify pre-prepare
        if (msg.view_number != state.get_current_view()) {
            return;
        }

        if (state.primary_of_view(msg.view_number) != msg.sender_id) {
            return;
        }

        if (!state.can_prepare(msg)) {
            return;
        }

        // Create PREPARE message
        Message prepare(Message::PREPARE,
                       msg.view_number,
                       msg.sequence_number,
                       msg.digest,
                       msg.content,
                       node_id);

        prepare.sign_message(node_id);

        // Byzantine nodes might send wrong digest
        if (byzantine && dist(rng) < 0.3) {
            stats.byzantine_actions++;
            prepare.digest = "WRONG_DIGEST";
        }

        if (broadcast_callback) {
            broadcast_callback(prepare);
            stats.messages_sent++;
        }

        state.add_prepare(msg);
    }

    void handle_prepare(const Message& msg) {
        state.add_prepare(msg);

        // Check if we have prepare certificate
        if (state.has_prepare_certificate(msg.sequence_number, msg.digest)) {
            // Create COMMIT message
            Message commit(Message::COMMIT,
                          msg.view_number,
                          msg.sequence_number,
                          msg.digest,
                          msg.content,
                          node_id);

            commit.sign_message(node_id);

            if (broadcast_callback) {
                broadcast_callback(commit);
                stats.messages_sent++;
            }
        }
    }

    void handle_commit(const Message& msg) {
        state.add_commit(msg);

        // Check if we have commit certificate
        if (state.has_commit_certificate(msg.sequence_number, msg.digest)) {
            // Check if we can execute
            if (state.can_execute(msg.sequence_number)) {
                execute_request(msg);
                state.mark_executed(msg.sequence_number);
                stats.requests_executed++;
            }
        }
    }

    void execute_request(const Message& msg) {
        std::cout << "Node " << node_id << " executing request: " 
                  << msg.sequence_number << " - " << msg.content << std::endl;

        // In real implementation, this would execute the state machine operation

        // Send reply to client
        Message reply(Message::REPLY,
                     msg.view_number,
                     msg.sequence_number,
                     msg.digest,
                     "EXECUTED: " + msg.content,
                     node_id);

        reply.client_id = msg.client_id;
        reply.sign_message(node_id);

        // Send to client (simplified)
        if (send_callback) {
            // Client ID 0 for simplicity
            send_callback(0, reply);
        }
    }

    void handle_view_change(const Message& msg) {
        // Start view change if we haven't already
        if (msg.view_number > state.get_current_view()) {
            state.start_view_change(msg.view_number);
            stats.view_changes++;
        }

        state.add_view_change(msg.view_number, msg.sender_id);

        // Check if we have enough view change messages
        if (state.has_view_change_certificate(msg.view_number)) {
            // New primary sends NEW-VIEW
            if (state.primary_of_view(msg.view_number) == node_id) {
                Message new_view(Message::NEW_VIEW,
                                msg.view_number,
                                0,  // sequence doesn't matter
                                "",
                                "NEW_VIEW_CONTENT",
                                node_id);

                new_view.sign_message(node_id);

                if (broadcast_callback) {
                    broadcast_callback(new_view);
                }
            }
        }
    }

    void handle_new_view(const Message& msg) {
        // Verify new view comes from primary of that view
        if (state.primary_of_view(msg.view_number) != msg.sender_id) {
            return;
        }

        // Update our view
        state.start_view_change(msg.view_number);

        std::cout << "Node " << node_id << " moving to view " 
                  << msg.view_number << std::endl;
    }

    const Stats& get_stats() const { return stats; }
    int get_id() const { return node_id; }
    bool is_byzantine() const { return byzantine; }
};

// ============================================================================
// PBFT NETWORK SIMULATION
// ============================================================================

class PBFTNetwork {
private:
    std::vector<std::unique_ptr<PBFTNode>> nodes;
    std::map<int, std::vector<Message>> message_queues;
    std::mutex network_mutex;

    int total_nodes;
    int byzantine_count;

    // Statistics
    struct NetworkStats {
        int total_messages = 0;
        int delivered_messages = 0;
        int dropped_messages = 0;
        std::map<int, int> messages_by_type;
        std::chrono::steady_clock::time_point start_time;
    } stats;

public:
    PBFTNetwork(int n, int byzantine) 
        : total_nodes(n), byzantine_count(byzantine) {

        if (n <= 3 * byzantine) {
            throw std::runtime_error("N must be > 3f for Byzantine tolerance");
        }

        // Create nodes
        for (int i = 0; i < n; i++) {
            bool is_byzantine = i < byzantine;
            auto node = std::make_unique<PBFTNode>(i, n, is_byzantine);

            // Set callbacks
            node->set_network_callbacks(
                [this, i](const Message& msg) {
                    // Broadcast
                    for (int j = 0; j < total_nodes; j++) {
                        if (j != i) {
                            deliver_message(j, msg);
                        }
                    }
                },
                [this, i](int target, const Message& msg) {
                    // Send to specific node
                    if (target >= 0 && target < total_nodes && target != i) {
                        deliver_message(target, msg);
                    }
                }
            );

            nodes.push_back(std::move(node));
        }

        stats.start_time = std::chrono::steady_clock::now();
        message_queues.resize(n);
    }

    void deliver_message(int target, const Message& msg) {
        std::lock_guard<std::mutex> lock(network_mutex);

        stats.total_messages++;
        stats.messages_by_type[msg.type]++;

        // Simulate network delays (0-50ms)
        std::this_thread::sleep_for(
            std::chrono::milliseconds(rand() % 50));

        // Simulate message loss (2%)
        if (rand() % 100 < 2) {
            stats.dropped_messages++;
            return;
        }

        message_queues[target].push_back(msg);
    }

    void process_messages() {
        std::lock_guard<std::mutex> lock(network_mutex);

        for (int i = 0; i < total_nodes; i++) {
            for (const auto& msg : message_queues[i]) {
                nodes[i]->receive_message(msg);
                stats.delivered_messages++;
            }
            message_queues[i].clear();
        }
    }

    void broadcast_client_request(const std::string& request) {
        Message msg(Message::REQUEST,
                   0,  // view
                   0,  // sequence (client doesn't know)
                   CryptographicHash::sha256(request),
                   request,
                   -1);  // client

        msg.client_id = "client_0";
        msg.sign_message(-1);

        // Send to all nodes
        for (int i = 0; i < total_nodes; i++) {
            deliver_message(i, msg);
        }
    }

    void run_consensus(int num_requests) {
        std::cout << "\nStarting PBFT consensus with " << total_nodes 
                  << " nodes (" << byzantine_count << " Byzantine)" << std::endl;

        // Start all nodes
        for (auto& node : nodes) {
            node->start();
        }

        // Send client requests
        for (int i = 0; i < num_requests; i++) {
            std::string request = "REQUEST_" + std::to_string(i) + 
                                 "_" + std::to_string(rand());

            std::cout << "\nSending request: " << request << std::endl;
            broadcast_client_request(request);

            // Process messages
            for (int j = 0; j < 5; j++) {  // Multiple rounds
                process_messages();
                std::this_thread::sleep_for(std::chrono::milliseconds(100));
            }
        }

        // Stop nodes
        for (auto& node : nodes) {
            node->stop();
        }

        print_statistics();
    }

    void print_statistics() {
        auto end_time = std::chrono::steady_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(
            end_time - stats.start_time);

        std::cout << "\n" << std::string(60, '=') << std::endl;
        std::cout << "PBFT SIMULATION STATISTICS" << std::endl;
        std::cout << std::string(60, '=') << std::endl;

        std::cout << "\nNetwork Statistics:" << std::endl;
        std::cout << "  Total messages: " << stats.total_messages << std::endl;
        std::cout << "  Delivered messages: " << stats.delivered_messages << std::endl;
        std::cout << "  Dropped messages: " << stats.dropped_messages 
                  << " (" << (stats.dropped_messages * 100.0 / stats.total_messages) 
                  << "%)" << std::endl;
        std::cout << "  Duration: " << duration.count() << "ms" << std::endl;

        std::cout << "\nMessage types:" << std::endl;
        for (auto& [type, count] : stats.messages_by_type) {
            std::string type_str;
            switch(type) {
                case Message::REQUEST: type_str = "REQUEST"; break;
                case Message::PREPREPARE: type_str = "PREPREPARE"; break;
                case Message::PREPARE: type_str = "PREPARE"; break;
                case Message::COMMIT: type_str = "COMMIT"; break;
                case Message::REPLY: type_str = "REPLY"; break;
                case Message::VIEW_CHANGE: type_str = "VIEW_CHANGE"; break;
                case Message::NEW_VIEW: type_str = "NEW_VIEW"; break;
            }
            std::cout << "  " << type_str << ": " << count << std::endl;
        }

        std::cout << "\nNode Statistics:" << std::endl;
        for (auto& node : nodes) {
            auto node_stats = node->get_stats();
            std::cout << "  Node " << node->get_id() 
                      << (node->is_byzantine() ? " (Byzantine)" : " (Honest)") 
                      << ":" << std::endl;
            std::cout << "    Messages received: " << node_stats.messages_received << std::endl;
            std::cout << "    Messages sent: " << node_stats.messages_sent << std::endl;
            std::cout << "    Requests executed: " << node_stats.requests_executed << std::endl;
            std::cout << "    Byzantine actions: " << node_stats.byzantine_actions << std::endl;
            std::cout << "    View changes: " << node_stats.view_changes << std::endl;
        }
    }
};

// ============================================================================
// BYZANTINE AGREEMENT PROOFS AND ANALYSIS
// ============================================================================

void prove_byzantine_agreement() {
    std::cout << "=" << std::string(70, '=') << std::endl;
    std::cout << "BYZANTINE AGREEMENT: MATHEMATICAL PROOFS" << std::endl;
    std::cout << "=" << std::string(70, '=') << std::endl;

    std::cout << "\nTheorem 1: With 3 generals and 1 traitor, agreement is impossible" << std::endl;
    std::cout << "Proof (by contradiction):" << std::endl;
    std::cout << "  1. Assume generals A, B, C where C is traitor." << std::endl;
    std::cout << "  2. A sends 'attack' to B and C." << std::endl;
    std::cout << "  3. C tells B that A said 'retreat' (lying)." << std::endl;
    std::cout << "  4. B hears 'attack' from A and 'retreat' from C." << std::endl;
    std::cout << "  5. B cannot determine who is lying: A or C?" << std::endl;
    std::cout << "  6. Therefore, B cannot decide consistently with A. ✓" << std::endl;

    std::cout << "\nTheorem 2: Agreement is possible iff N ≥ 3f + 1" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  Necessity (N ≥ 3f + 1):" << std::endl;
    std::cout << "    1. Divide N nodes into 3 groups: honest (H), faulty (F), slow (S)." << std::endl;
    std::cout << "    2. Worst case: |F| = f, |S| = f (non-responding)." << std::endl;
    std::cout << "    3. Need |H| ≥ f + 1 to outnumber faulty nodes." << std::endl;
    std::cout << "    4. Therefore: N = |H| + |F| + |S| ≥ (f + 1) + f + f = 3f + 1. ✓" << std::endl;

    std::cout << "\n  Sufficiency (algorithm exists for N ≥ 3f + 1):" << std::endl;
    std::cout << "    1. PBFT protocol provides safety and liveness." << std::endl;
    std::cout << "    2. Safety: Honest nodes agree on order of requests." << std::endl;
    std::cout << "    3. Liveness: Client eventually receives reply." << std::endl;
    std::cout << "    4. Proof via quorum intersection: any two quorums of size 2f+1" << std::endl;
    std::cout << "       intersect in at least f+1 honest nodes. ✓" << std::endl;

    std::cout << "\nTheorem 3: Optimal resilience is f < N/3" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  1. Assume f ≥ N/3." << std::endl;
    std::cout << "  2. Then N ≤ 3f." << std::endl;
    std::cout << "  3. From Theorem 2, agreement is impossible." << std::endl;
    std::cout << "  4. Therefore, maximum tolerable faults is f < N/3. ✓" << std::endl;

    std::cout << "\nTheorem 4: PBFT provides linearizability" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  1. All operations are totally ordered by sequence numbers." << std::endl;
    std::cout << "  2. Commit certificates ensure all honest nodes execute in same order." << std::endl;
    std::cout << "  3. View changes preserve sequence number ordering." << std::endl;
    std::cout << "  4. Therefore, execution is equivalent to a centralized system. ✓" << std::endl;
}

void analyze_byzantine_scenarios() {
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "BYZANTINE FAILURE SCENARIO ANALYSIS" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    // Test different configurations
    struct Scenario {
        int total_nodes;
        int byzantine_nodes;
        bool should_succeed;
        std::string description;
    };

    std::vector<Scenario> scenarios = {
        {4, 1, true, "Minimal configuration (N=4, f=1)"},
        {7, 2, true, "Typical configuration (N=7, f=2)"},
        {10, 3, true, "Large configuration (N=10, f=3)"},
        {3, 1, false, "Impossible case (N=3, f=1)"},
        {6, 2, true, "Boundary case (N=6, f=2)"},
        {5, 2, false, "Too many faults (N=5, f=2)"},
    };

    std::cout << "\nScenario Analysis:" << std::endl;
    std::cout << std::string(80, '-') << std::endl;
    std::cout << std::setw(20) << "Configuration" 
              << std::setw(15) << "N ≥ 3f+1" 
              << std::setw(15) << "Expected" 
              << std::setw(30) << "Description" << std::endl;
    std::cout << std::string(80, '-') << std::endl;

    for (const auto& scenario : scenarios) {
        bool condition = scenario.total_nodes >= 3 * scenario.byzantine_nodes + 1;

        std::cout << std::setw(10) << "N=" + std::to_string(scenario.total_nodes)
                  << std::setw(10) << "f=" + std::to_string(scenario.byzantine_nodes)
                  << std::setw(15) << (condition ? "Yes" : "No")
                  << std::setw(15) << (scenario.should_succeed ? "Success" : "Failure")
                  << std::setw(30) << scenario.description << std::endl;
    }

    // Mathematical analysis
    std::cout << "\nMathematical Analysis:" << std::endl;
    std::cout << std::string(80, '-') << std::endl;

    for (int N = 3; N <= 10; N++) {
        int max_f = (N - 1) / 3;
        double resilience = max_f * 100.0 / N;

        std::cout << "N=" << N << ": Max f=" << max_f 
                  << " (" << resilience << "% nodes can be Byzantine)"
                  << std::endl;
    }
}

void demonstrate_pbft_phases() {
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "PBFT PROTOCOL PHASES DEMONSTRATION" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    // Create a minimal PBFT network
    int N = 4;
    int f = 1;

    std::cout << "\nNormal Case Operation (N=" << N << ", f=" << f << "):" << std::endl;
    std::cout << "1. Client sends request to all replicas" << std::endl;
    std::cout << "2. Primary (replica 0) assigns sequence number" << std::endl;
    std::cout << "3. Primary broadcasts PRE-PREPARE" << std::endl;
    std::cout << "4. Replicas broadcast PREPARE after verifying" << std::endl;
    std::cout << "5. After 2f PREPARE messages, replicas broadcast COMMIT" << std::endl;
    std::cout << "6. After 2f+1 COMMIT messages, replicas execute request" << std::endl;
    std::cout << "7. Replicas send REPLY to client" << std::endl;
    std::cout << "8. Client waits for f+1 matching replies" << std::endl;

    std::cout << "\nView Change Protocol:" << std::endl;
    std::cout << "1. Replica suspects primary is faulty (timeout)" << std::endl;
    std::cout << "2. Replica broadcasts VIEW-CHANGE to new view" << std::endl;
    std::cout << "3. After 2f+1 VIEW-CHANGE messages, new primary is elected" << std::endl;
    std::cout << "4. New primary broadcasts NEW-VIEW with checkpoint" << std::endl;
    std::cout << "5. Replicas move to new view and resume operation" << std::endl;

    std::cout << "\nSafety Argument:" << std::endl;
    std::cout << "- Any two quorums of size 2f+1 intersect in at least f+1 honest nodes" << std::endl;
    std::cout << "- These honest nodes ensure consistent ordering" << std::endl;
    std::cout << "- View changes preserve ordering through sequence numbers" << std::endl;

    std::cout << "\nLiveness Argument:" << std::endl;
    std::cout << "- Eventually, a view with non-faulty primary will last long enough" << std::endl;
    std::cout << "- Timeouts ensure progress despite faulty primaries" << std::endl;
    std::cout << "- GST (Global Stabilization Time) assumption" << std::endl;
}

int main() {
    std::cout << "BYZANTINE FAULT TOLERANCE: COMPLETE IMPLEMENTATION AND ANALYSIS" << std::endl;

    // Run mathematical proofs
    prove_byzantine_agreement();

    // Analyze scenarios
    analyze_byzantine_scenarios();

    // Demonstrate PBFT phases
    demonstrate_pbft_phases();

    // Run simulations
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "PBFT SIMULATIONS" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    try {
        // Test 1: Normal operation
        std::cout << "\nTest 1: Normal operation (N=4, f=0)" << std::endl;
        PBFTNetwork network1(4, 0);
        network1.run_consensus(3);

        // Test 2: With Byzantine nodes
        std::cout << "\n\nTest 2: With Byzantine nodes (N=7, f=2)" << std::endl;
        PBFTNetwork network2(7, 2);
        network2.run_consensus(3);

        // Test 3: Impossible case (should throw)
        std::cout << "\n\nTest 3: Impossible case (N=3, f=1)" << std::endl;
        try {
            PBFTNetwork network3(3, 1);
            network3.run_consensus(1);
        } catch (const std::runtime_error& e) {
            std::cout << "Expected error: " << e.what() << std::endl;
        }

    } catch (const std::exception& e) {
        std::cerr << "Error: " << e.what() << std::endl;
    }

    return 0;
}

6. Threshold Cryptography: Shamir's Secret Sharing

Mathematical Foundation

Shamir's Secret Sharing is based on polynomial interpolation over finite fields:

Theorem: Given any t points in the plane with distinct x-coordinates, there exists exactly one polynomial of degree t-1 that passes through all points.

Algorithm:

Choose a prime p > secret
Create polynomial: f(x) = s + a₁x + a₂x² + ... + aₜ₋₁xᵗ⁻¹ mod p
Share point (i, f(i)) with party i
Any t points can reconstruct via Lagrange interpolation

Complete Python Implementation

import random
from typing import List, Tuple
import hashlib
from dataclasses import dataclass
from functools import lru_cache
import sympy as sp

class ShamirSecretSharing:
    """
    Complete implementation of Shamir's Secret Sharing
    """

    def __init__(self, prime: int = 2**127 - 1):
        """
        Initialize with a large prime
        """
        self.prime = prime

    def generate_shares(self, secret: int, n: int, t: int) -> List[Tuple[int, int]]:
        """
        Generate n shares, any t of which can reconstruct the secret

        Args:
            secret: The secret to share (integer)
            n: Total number of shares
            t: Threshold (minimum shares needed)

        Returns:
            List of (x, y) pairs
        """
        if t > n:
            raise ValueError("Threshold cannot exceed total shares")

        if secret >= self.prime:
            raise ValueError("Secret must be less than prime")

        # Generate random coefficients for polynomial of degree t-1
        coefficients = [secret] + [random.randint(1, self.prime - 1) 
                                  for _ in range(t - 1)]

        # Evaluate polynomial at points 1, 2, ..., n
        shares = []
        for x in range(1, n + 1):
            y = self._evaluate_polynomial(coefficients, x)
            shares.append((x, y))

        return shares

    def _evaluate_polynomial(self, coeffs: List[int], x: int) -> int:
        """Evaluate polynomial using Horner's method"""
        result = 0
        for coeff in reversed(coeffs):
            result = (result * x + coeff) % self.prime
        return result

    def reconstruct_secret(self, shares: List[Tuple[int, int]]) -> int:
        """
        Reconstruct secret using Lagrange interpolation

        Args:
            shares: List of at least t shares

        Returns:
            Reconstructed secret
        """
        if not shares:
            raise ValueError("No shares provided")

        # Lagrange interpolation to compute f(0)
        secret = 0

        for i, (xi, yi) in enumerate(shares):
            # Compute Lagrange basis polynomial L_i(0)
            numerator = 1
            denominator = 1

            for j, (xj, _) in enumerate(shares):
                if i == j:
                    continue

                numerator = (numerator * (-xj)) % self.prime
                denominator = (denominator * (xi - xj)) % self.prime

            # L_i(0) = numerator / denominator
            li = (numerator * self._mod_inverse(denominator)) % self.prime

            # Add contribution of this share
            secret = (secret + yi * li) % self.prime

        return secret

    def _mod_inverse(self, a: int) -> int:
        """Compute modular inverse using extended Euclidean algorithm"""
        return pow(a, -1, self.prime)  # Python 3.8+

    def verify_share(self, share: Tuple[int, int], commitment: List[int]) -> bool:
        """
        Verify share validity using Feldman's VSS
        """
        x, y = share

        # g^y should equal product of commitments^ (x^i)
        # Simplified verification
        lhs = pow(2, y, self.prime)  # Using 2 as generator

        rhs = 1
        for i, comm in enumerate(commitment):
            rhs = (rhs * pow(comm, x**i, self.prime)) % self.prime

        return lhs == rhs

class ThresholdSignature:
    """
    Threshold signatures using Shamir's Secret Sharing
    """

    def __init__(self, n: int, t: int):
        self.n = n
        self.t = t
        self.sss = ShamirSecretSharing()

        # Generate key shares
        self.private_key = random.randint(1, 2**256)
        self.shares = self.sss.generate_shares(self.private_key, n, t)

        # Public commitments for verification
        self.commitments = self._generate_commitments()

    def _generate_commitments(self) -> List[int]:
        """Generate Feldman commitments for verification"""
        # For simplicity, use g = 2
        g = 2
        prime = self.sss.prime

        commitments = []
        for i in range(self.t):
            # Commitment for coefficient i
            comm = pow(g, random.randint(1, prime - 1), prime)
            commitments.append(comm)

        return commitments

    def sign_share(self, message: str, share_idx: int) -> Tuple[int, int]:
        """
        Generate signature share for a message

        Returns: (share_index, signature_share)
        """
        x, y = self.shares[share_idx]

        # Hash message
        msg_hash = int(hashlib.sha256(message.encode()).hexdigest(), 16)
        msg_hash = msg_hash % self.sss.prime

        # Generate signature share
        # Simplified: s_i = y * H(m) mod p
        sig_share = (y * msg_hash) % self.sss.prime

        return x, sig_share

    def combine_signatures(self, sig_shares: List[Tuple[int, int]]) -> int:
        """
        Combine signature shares using Lagrange interpolation

        Returns: Complete signature
        """
        # Reconstruct signature using same method as secret sharing
        signature = 0

        for i, (xi, si) in enumerate(sig_shares):
            # Compute Lagrange basis polynomial L_i(0)
            numerator = 1
            denominator = 1

            for j, (xj, _) in enumerate(sig_shares):
                if i == j:
                    continue

                numerator = (numerator * (-xj)) % self.sss.prime
                denominator = (denominator * (xi - xj)) % self.sss.prime

            li = (numerator * self.sss._mod_inverse(denominator)) % self.sss.prime

            signature = (signature + si * li) % self.sss.prime

        return signature

    def verify_signature(self, message: str, signature: int) -> bool:
        """
        Verify threshold signature

        Returns: True if signature is valid
        """
        # Hash message
        msg_hash = int(hashlib.sha256(message.encode()).hexdigest(), 16)
        msg_hash = msg_hash % self.sss.prime

        # Compute expected: g^signature mod p
        g = 2
        lhs = pow(g, signature, self.sss.prime)

        # Compute: y^H(m) mod p where y = g^private_key
        y = pow(g, self.private_key, self.sss.prime)
        rhs = pow(y, msg_hash, self.sss.prime)

        return lhs == rhs

class DistributedKeyGeneration:
    """
    Distributed Key Generation without trusted dealer
    """

    def __init__(self, n: int, t: int):
        self.n = n
        self.t = t
        self.sss = ShamirSecretSharing()

        # Each party will generate their own polynomial
        self.parties = []

    class Party:
        def __init__(self, party_id: int, n: int, t: int, sss):
            self.id = party_id
            self.n = n
            self.t = t
            self.sss = sss

            # Generate random secret
            self.secret = random.randint(1, sss.prime - 1)

            # Create shares for other parties
            self.shares = sss.generate_shares(self.secret, n, t)

            # Generate commitments for verification
            self.commitments = self._generate_commitments()

        def _generate_commitments(self):
            """Generate commitments to polynomial coefficients"""
            g = 2
            prime = self.sss.prime

            # For simplicity, just commit to the secret
            return [pow(g, self.secret, prime)]

        def get_share_for(self, party_id: int) -> Tuple[int, int]:
            """Get share for specific party"""
            return self.shares[party_id - 1]

    def run_protocol(self):
        """Run DKG protocol"""
        print("Starting Distributed Key Generation...")

        # Create parties
        self.parties = [self.Party(i, self.n, self.t, self.sss) 
                       for i in range(1, self.n + 1)]

        # Phase 1: Each party sends shares to others
        print("\nPhase 1: Distributing shares...")
        received_shares = {i: [] for i in range(1, self.n + 1)}

        for party in self.parties:
            for other_id in range(1, self.n + 1):
                if other_id == party.id:
                    continue

                share = party.get_share_for(other_id)
                received_shares[other_id].append((party.id, share))

        # Phase 2: Verify shares
        print("\nPhase 2: Verifying shares...")
        valid_shares = {i: [] for i in range(1, self.n + 1)}

        for party_id in range(1, self.n + 1):
            for share_giver_id, share in received_shares[party_id]:
                # Verify share using commitments
                x, y = share
                giver = self.parties[share_giver_id - 1]

                # Simplified verification
                if y < self.sss.prime:  # Basic check
                    valid_shares[party_id].append(share)

        # Phase 3: Compute final key shares
        print("\nPhase 3: Computing final key shares...")
        final_shares = []

        for party_id in range(1, self.n + 1):
            # Each party sums received shares
            party_shares = valid_shares[party_id]
            if len(party_shares) < self.t:
                print(f"Party {party_id}: insufficient valid shares")
                continue

            # Sum y-values for same x
            x = party_id
            y_sum = 0

            for _, (_, y) in party_shares:
                y_sum = (y_sum + y) % self.sss.prime

            final_shares.append((x, y_sum))

        # The sum of secrets is the distributed key
        return final_shares

    def reconstruct_key(self, shares: List[Tuple[int, int]]) -> int:
        """Reconstruct distributed key from shares"""
        return self.sss.reconstruct_secret(shares)

def demonstrate_threshold_crypto():
    """
    Complete demonstration of threshold cryptography
    """
    print("="*70)
    print("THRESHOLD CRYPTOGRAPHY COMPLETE DEMONSTRATION")
    print("="*70)

    # Parameters
    n = 5  # Total shares
    t = 3  # Threshold

    # 1. Basic Secret Sharing
    print("\n1. Shamir's Secret Sharing:")
    print("-"*40)

    sss = ShamirSecretSharing()
    secret = 123456789

    print(f"Original secret: {secret}")

    # Generate shares
    shares = sss.generate_shares(secret, n, t)
    print(f"\nGenerated {n} shares (need {t} to reconstruct):")
    for i, (x, y) in enumerate(shares):
        print(f"  Share {i+1}: ({x}, {y})")

    # Reconstruct with threshold
    print(f"\nReconstructing with {t} shares...")
    selected_shares = shares[:t]
    reconstructed = sss.reconstruct_secret(selected_shares)
    print(f"Reconstructed secret: {reconstructed}")
    print(f"Match: {'✓' if reconstructed == secret else '✗'}")

    # Try with insufficient shares
    print(f"\nTrying with {t-1} shares (should fail)...")
    try:
        insufficient = shares[:t-1]
        sss.reconstruct_secret(insufficient)
        print("Unexpected success!")
    except Exception as e:
        print(f"Expected error: {e}")

    # 2. Threshold Signatures
    print("\n\n2. Threshold Signatures:")
    print("-"*40)

    ts = ThresholdSignature(n, t)
    message = "Hello, threshold crypto!"

    print(f"Message: {message}")

    # Generate signature shares
    print(f"\nGenerating signature shares from {t} parties...")
    sig_shares = []
    for i in range(t):
        x, sig_share = ts.sign_share(message, i)
        sig_shares.append((x, sig_share))
        print(f"  Party {i}: share = {sig_share}")

    # Combine signatures
    print("\nCombining signature shares...")
    full_signature = ts.combine_signatures(sig_shares)
    print(f"Full signature: {full_signature}")

    # Verify signature
    print("\nVerifying signature...")
    is_valid = ts.verify_signature(message, full_signature)
    print(f"Signature valid: {'✓' if is_valid else '✗'}")

    # 3. Distributed Key Generation
    print("\n\n3. Distributed Key Generation (no trusted dealer):")
    print("-"*40)

    dkg = DistributedKeyGeneration(n, t)
    final_shares = dkg.run_protocol()

    print(f"\nGenerated {len(final_shares)} final shares")

    # Reconstruct distributed key
    if len(final_shares) >= t:
        distributed_key = dkg.reconstruct_key(final_shares[:t])
        print(f"Distributed key: {distributed_key}")

    # 4. Security Analysis
    print("\n\n4. Security Analysis:")
    print("-"*40)

    print("\nInformation Theoretic Security:")
    print("  - With t-1 shares, zero information about secret")
    print("  - Security doesn't depend on computational assumptions")
    print("  - Perfect secrecy for finite fields")

    print("\nProperties:")
    print(f"  - Additive homomorphism: f(x) + g(x) = (f+g)(x)")
    print(f"  - Any subset of size t can reconstruct")
    print(f"  - Any subset of size < t learns nothing")

    print("\nApplications:")
    print("  - Distributed custody of crypto assets")
    print("  - Secure multi-party computation")
    print("  - Byzantine fault tolerance")
    print("  - Password recovery systems")

def mathematical_proofs():
    """
    Mathematical proofs for threshold cryptography
    """
    print("\n" + "="*70)
    print("MATHEMATICAL PROOFS FOR THRESHOLD CRYPTOGRAPHY")
    print("="*70)

    print("\nTheorem 1: Lagrange Interpolation Uniqueness")
    print("Proof:")
    print("  Given: Points (x₁, y₁), ..., (xₜ, yₜ) with distinct xᵢ")
    print("  Want: Polynomial f of degree ≤ t-1 with f(xᵢ) = yᵢ")
    print("  Construct Lagrange basis polynomials:")
    print("    Lᵢ(x) = ∏_{j≠i} (x - xⱼ) / (xᵢ - xⱼ)")
    print("  Then f(x) = Σ yᵢ Lᵢ(x)")
    print("  f has degree ≤ t-1 and f(xᵢ) = yᵢ")
    print("  Uniqueness: If g also satisfies, then f-g has t roots")
    print("  but degree ≤ t-1, so f-g ≡ 0 ⇒ f = g ✓")

    print("\nTheorem 2: Information Theoretic Security")
    print("Proof:")
    print("  Given any t-1 shares and any secret s, ∃ polynomial f")
    print("  with f(0) = s passing through given points")
    print("  Specifically: choose random aₜ₋₁, solve for others")
    print("  Thus probability distribution of secret is uniform")
    print("  given t-1 shares ⇒ perfect secrecy ✓")

    print("\nTheorem 3: Linear Homomorphism")
    print("Proof:")
    print("  Let f, g be sharing polynomials for secrets s, s'")
    print("  Then h(x) = f(x) + g(x) is polynomial of same degree")
    print("  with h(0) = s + s'")
    print("  Shares are (xᵢ, f(xᵢ) + g(xᵢ))")
    print("  Thus secret sharing preserves addition ✓")

def analyze_threshold_security():
    """
    Analyze security of threshold schemes
    """
    print("\n" + "="*70)
    print("SECURITY ANALYSIS OF THRESHOLD SCHEMES")
    print("="*70)

    # Different configurations
    configurations = [
        (3, 2),  # 2-of-3
        (5, 3),  # 3-of-5
        (7, 4),  # 4-of-7
        (10, 6), # 6-of-10
    ]

    print("\nConfiguration Analysis:")
    print("-"*80)
    print(f"{'Scheme':<10} {'Shares Needed':<15} {'Total Shares':<15} "
          f"{'Security Level':<15} {'Redundancy':<15}")
    print("-"*80)

    for total, threshold in configurations:
        # Security increases with threshold
        security = 2**(threshold * 32)  # Approximate

        # Redundancy = total / threshold
        redundancy = total / threshold

        print(f"{threshold}-of-{total}:{threshold:<15} {total:<15} "
              f"{security:<15.2e} {redundancy:<15.2f}")

    # Attack scenarios
    print("\nAttack Scenarios:")
    print("-"*80)

    scenarios = [
        ("Brute force", "Try all possible secrets", "Exponential in field size"),
        ("Share theft", "Steal t-1 shares", "Still secure"),
        ("Malicious dealer", "Distribute invalid shares", "Use VSS"),
        ("Network attacks", "Intercept shares", "Use encryption"),
    ]

    for name, description, defense in scenarios:
        print(f"{name:<20} {description:<30} {defense:<30}")

if __name__ == "__main__":
    demonstrate_threshold_crypto()
    mathematical_proofs()
    analyze_threshold_security()

7. Formal Verification of Protocols: TLA+ and Model Checking

Mathematical Foundations of Formal Verification

Formal verification uses mathematical logic to prove correctness properties of systems:

Temporal Logic (LTL/CTL):

Linear Temporal Logic (LTL): Specifies properties over single execution paths

  ◇P = Eventually P (P will eventually be true)
  □P = Always P (P is always true)
  P U Q = P until Q (P holds until Q becomes true)

Computation Tree Logic (CTL): Specifies properties over computation trees

  ∃◇P = There exists a path where P eventually holds
  ∀□P = For all paths, P always holds

The TLA+ Approach:
TLA+ (Temporal Logic of Actions) combines:

Actions: State transitions described as mathematical formulas
Temporal Operators: To specify liveness and safety properties
Refinement: To prove implementation matches specification

Complete TLA+ Specification for Paxos Consensus

---------------------------- MODULE Paxos ----------------------------
EXTENDS Integers, Sequences, FiniteSets, TLC

CONSTANT Acceptor, Proposer, Value
ASSUME Acceptor \cap Proposer = {} \* Disjoint sets
ASSUME Cardinality(Acceptor) > 0 \* At least one acceptor
ASSUME Cardinality(Proposer) > 0 \* At least one proposer

VARIABLES
    ballot,           \* Current ballot number (proposer -> nat)
    accepted,         \* Accepted proposals (acceptor -> [ballot, value])
    proposed,         \* Proposed values (proposer -> value)
    decided           \* Decided values (value or nil)

TypeInvariant ==
    /\ ballot \in [Proposer -> Nat]
    /\ accepted \in [Acceptor -> SUBSET (Nat \times Value)]
    /\ proposed \in [Proposer -> Value \union {Nil}]
    /\ decided \in Value \union {Nil}

(*---------------------------------------------------------------
    Phase 1a: Proposer sends prepare request
----------------------------------------------------------------*)
SendPrepare(proposer) ==
    /\ proposed[proposer] = Nil
    /\ ballot' = [ballot EXCEPT ![proposer] = @ + 1]
    /\ UNCHANGED <<accepted, proposed, decided>>

(*---------------------------------------------------------------
    Phase 1b: Acceptor responds to prepare
----------------------------------------------------------------*)
ReceivePrepare(proposer, acceptor) ==
    /\ \E b \in Nat : \* There exists some ballot
        /\ b > ballot[proposer]
        /\ \A a \in Acceptor : \* No higher ballot accepted
            \A <b2, v2> \in accepted[a] : b2 < b
    /\ accepted' = [accepted EXCEPT ![acceptor] = @ \union {<ballot[proposer], Nil>}]
    /\ UNCHANGED <<ballot, proposed, decided>>

(*---------------------------------------------------------------
    Phase 2a: Proposer sends accept request
----------------------------------------------------------------*)
SendAccept(proposer, value) ==
    /\ proposed[proposer] = Nil
    /\ \E acceptor \in Acceptor : \* Majority promises
        LET promises == {a \in Acceptor : <ballot[proposer], Nil> \in accepted[a]}
        IN Cardinality(promises) > Cardinality(Acceptor) \div 2
    /\ proposed' = [proposed EXCEPT ![proposer] = value]
    /\ UNCHANGED <<ballot, accepted, decided>>

(*---------------------------------------------------------------
    Phase 2b: Acceptor accepts proposal
----------------------------------------------------------------*)
ReceiveAccept(proposer, acceptor, value) ==
    /\ proposed[proposer] = value
    /\ \E <b, _> \in accepted[acceptor] : b = ballot[proposer]
    /\ accepted' = [accepted EXCEPT ![acceptor] = 
                    @ \union {<ballot[proposer], value>}]
    /\ UNCHANGED <<ballot, proposed, decided>>

(*---------------------------------------------------------------
    Learn decision
----------------------------------------------------------------*)
LearnDecision(value) ==
    /\ \E acceptor \in Acceptor : \* Majority accepted
        LET accepts == {a \in Acceptor : \E b : <b, value> \in accepted[a]}
        IN Cardinality(accepts) > Cardinality(Acceptor) \div 2
    /\ decided' = value
    /\ UNCHANGED <<ballot, accepted, proposed>>

(*---------------------------------------------------------------
    Next-state relation
----------------------------------------------------------------*)
Next ==
    \/ \E p \in Proposer : SendPrepare(p)
    \/ \E p \in Proposer, a \in Acceptor : ReceivePrepare(p, a)
    \/ \E p \in Proposer, v \in Value : SendAccept(p, v)
    \/ \E p \in Proposer, a \in Acceptor, v \in Value : ReceiveAccept(p, a, v)
    \/ \E v \in Value : LearnDecision(v)

(*---------------------------------------------------------------
    Initial state
----------------------------------------------------------------*)
Init ==
    /\ ballot = [p \in Proposer |-> 0]
    /\ accepted = [a \in Acceptor |-> {}]
    /\ proposed = [p \in Proposer |-> Nil]
    /\ decided = Nil

(*---------------------------------------------------------------
    Safety properties
----------------------------------------------------------------*)
Safety ==
    /\ \A v1, v2 \in Value : \* Agreement: only one value decided
        (decided = v1 /\ decided = v2) => (v1 = v2)
    /\ \A a \in Acceptor : \* Consistency of accepted values
        \A <b1, v1>, <b2, v2> \in accepted[a] :
            (b1 = b2) => (v1 = v2)

(*---------------------------------------------------------------
    Liveness properties
----------------------------------------------------------------*)
Liveness ==
    /\ \E v \in Value : \* Validity: only proposed values can be decided
        (decided = v) => (\E p \in Proposer : proposed[p] = v)
    /\ <>(\E v \in Value : decided = v)  \* Termination: eventually decide

(*---------------------------------------------------------------
    Complete specification
----------------------------------------------------------------*)
Spec == Init /\ [][Next]_<<ballot, accepted, proposed, decided>> /\ Liveness

THEOREM Spec => []Safety  \* Safety is invariant

=============================================================================

Complete Python Implementation: Model Checking and Verification

import itertools
from typing import Set, Dict, List, Tuple, Optional, Any
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict
import time
import random
from functools import lru_cache

class LTLFormula:
    """
    Linear Temporal Logic formula implementation
    """

    def __init__(self, formula_type: str, *args):
        self.type = formula_type
        self.args = args

    def evaluate(self, trace: List[Dict[str, Any]], position: int = 0) -> bool:
        """Evaluate LTL formula on trace starting at position"""
        if self.type == "atomic":
            prop_name = self.args[0]
            return prop_name in trace[position]

        elif self.type == "not":
            return not self.args[0].evaluate(trace, position)

        elif self.type == "and":
            return all(arg.evaluate(trace, position) for arg in self.args)

        elif self.type == "or":
            return any(arg.evaluate(trace, position) for arg in self.args)

        elif self.type == "implies":
            left, right = self.args
            return (not left.evaluate(trace, position)) or right.evaluate(trace, position)

        elif self.type == "always":
            subformula = self.args[0]
            return all(subformula.evaluate(trace, i) for i in range(position, len(trace)))

        elif self.type == "eventually":
            subformula = self.args[0]
            return any(subformula.evaluate(trace, i) for i in range(position, len(trace)))

        elif self.type == "until":
            left, right = self.args
            for i in range(position, len(trace)):
                if right.evaluate(trace, i):
                    return True
                if not left.evaluate(trace, i):
                    return False
            return False

        elif self.type == "next":
            subformula = self.args[0]
            if position + 1 < len(trace):
                return subformula.evaluate(trace, position + 1)
            return False

        else:
            raise ValueError(f"Unknown formula type: {self.type}")

    def to_str(self) -> str:
        if self.type == "atomic":
            return self.args[0]
        elif self.type == "not":
            return f"¬({self.args[0].to_str()})"
        elif self.type == "and":
            return f"({' ∧ '.join(arg.to_str() for arg in self.args)})"
        elif self.type == "or":
            return f"({' ∨ '.join(arg.to_str() for arg in self.args)})"
        elif self.type == "implies":
            return f"({self.args[0].to_str()} → {self.args[1].to_str()})"
        elif self.type == "always":
            return f"□({self.args[0].to_str()})"
        elif self.type == "eventually":
            return f"◇({self.args[0].to_str()})"
        elif self.type == "until":
            return f"({self.args[0].to_str()} U {self.args[1].to_str()})"
        elif self.type == "next":
            return f"○({self.args[0].to_str()})"

class StateMachine:
    """
    Finite state machine for protocol verification
    """

    def __init__(self, states: Set[str], 
                 initial_state: str,
                 transitions: Dict[str, List[str]]):
        self.states = states
        self.initial_state = initial_state
        self.transitions = transitions

        # Atomic propositions for each state
        self.state_props = {state: set() for state in states}

        # Cache for reachable states
        self._reachable_cache = {}

    def add_proposition(self, state: str, prop: str):
        """Add atomic proposition to state"""
        self.state_props[state].add(prop)

    def get_successors(self, state: str) -> List[str]:
        """Get all possible next states"""
        return self.transitions.get(state, [])

    def get_trace_props(self, trace: List[str]) -> List[Set[str]]:
        """Get propositions for each state in trace"""
        return [self.state_props[state] for state in trace]

    def check_ltl(self, formula: LTLFormula, max_depth: int = 10) -> Tuple[bool, Optional[List[str]]]:
        """
        Model check LTL formula using DFS
        Returns: (satisfied, counterexample_trace)
        """
        visited = set()

        def dfs(state: str, depth: int, trace: List[str]) -> Tuple[bool, Optional[List[str]]]:
            if depth > max_depth:
                return True, None  # Assume satisfied for bounded model checking

            trace_key = (state, depth)
            if trace_key in visited:
                return True, None

            visited.add(trace_key)
            current_trace = trace + [state]

            # Get propositions for current trace
            trace_props = self.get_trace_props(current_trace)

            # Check if current trace violates formula
            if not formula.evaluate(trace_props, 0):
                return False, current_trace

            # Check all successor states
            for next_state in self.get_successors(state):
                satisfied, counterexample = dfs(next_state, depth + 1, current_trace)
                if not satisfied:
                    return False, counterexample

            return True, None

        return dfs(self.initial_state, 0, [])

    def check_invariant(self, invariant: LTLFormula) -> Tuple[bool, Optional[List[str]]]:
        """
        Check state invariant (□P)
        """
        return self.check_ltl(LTLFormula("always", invariant))

    def check_liveness(self, liveness: LTLFormula) -> Tuple[bool, Optional[List[str]]]:
        """
        Check liveness property (◇P)
        """
        return self.check_ltl(LTLFormula("eventually", liveness))

    def get_all_traces(self, max_length: int) -> List[List[str]]:
        """Generate all possible traces up to max length"""
        traces = []

        def dfs(state: str, depth: int, trace: List[str]):
            if depth >= max_length:
                traces.append(trace.copy())
                return

            current_trace = trace + [state]
            traces.append(current_trace.copy())

            for next_state in self.get_successors(state):
                dfs(next_state, depth + 1, current_trace)

        dfs(self.initial_state, 0, [])
        return traces

class PaxosModelChecker:
    """
    Complete model checker for Paxos consensus protocol
    """

    def __init__(self, num_proposers: int = 2, num_acceptors: int = 3):
        self.num_proposers = num_proposers
        self.num_acceptors = num_acceptors

        # State representation
        self.states = set()
        self.transitions = defaultdict(list)

        # Build state machine
        self._build_state_space()

    def _state_to_str(self, state: Dict) -> str:
        """Convert state dictionary to string representation"""
        parts = []

        # Ballot numbers
        for i in range(self.num_proposers):
            parts.append(f"B{i}={state.get(f'ballot_{i}', 0)}")

        # Accepted values
        for i in range(self.num_acceptors):
            accepted = state.get(f'accepted_{i}', (0, None))
            parts.append(f"A{i}=({accepted[0]},{accepted[1]})")

        # Proposed values
        for i in range(self.num_proposers):
            proposed = state.get(f'proposed_{i}', None)
            parts.append(f"P{i}={proposed}")

        # Decision
        decision = state.get('decision', None)
        parts.append(f"D={decision}")

        return "|".join(parts)

    def _build_state_space(self):
        """Build complete state space for Paxos"""
        # Generate all possible states
        # This is simplified - real Paxos has infinite state space

        # Initialize with empty state
        initial_state = {}
        for i in range(self.num_proposers):
            initial_state[f'ballot_{i}'] = 0
            initial_state[f'proposed_{i}'] = None

        for i in range(self.num_acceptors):
            initial_state[f'accepted_{i}'] = (0, None)

        initial_state['decision'] = None

        initial_str = self._state_to_str(initial_state)
        self.states.add(initial_str)

        # BFS to explore state space (limited)
        queue = [initial_state]
        visited = {initial_str}

        max_states = 1000  # Limit for tractability
        state_count = 0

        while queue and state_count < max_states:
            state = queue.pop(0)
            state_str = self._state_to_str(state)
            state_count += 1

            # Generate all possible next states
            next_states = self._get_next_states(state)

            for next_state in next_states:
                next_str = self._state_to_str(next_state)

                if next_str not in visited:
                    visited.add(next_str)
                    self.states.add(next_str)
                    queue.append(next_state)

                # Add transition
                if next_str not in self.transitions[state_str]:
                    self.transitions[state_str].append(next_str)

    def _get_next_states(self, state: Dict) -> List[Dict]:
        """Generate all possible next states from current state"""
        next_states = []

        # Phase 1a: Proposer sends prepare
        for i in range(self.num_proposers):
            if state.get(f'proposed_{i}') is None:
                new_state = state.copy()
                new_state[f'ballot_{i}'] = state.get(f'ballot_{i}', 0) + 1
                next_states.append(new_state)

        # Phase 1b: Acceptor responds to prepare
        for i in range(self.num_acceptors):
            current_ballot, current_value = state.get(f'accepted_{i}', (0, None))

            # Find highest ballot from proposers
            max_ballot = max(state.get(f'ballot_{j}', 0) 
                           for j in range(self.num_proposers))

            if max_ballot > current_ballot:
                new_state = state.copy()
                new_state[f'accepted_{i}'] = (max_ballot, None)
                next_states.append(new_state)

        # Phase 2a: Proposer sends accept
        for i in range(self.num_proposers):
            if state.get(f'proposed_{i}') is None:
                # Check if proposer has majority promises
                ballot = state.get(f'ballot_{i}', 0)
                promises = 0

                for j in range(self.num_acceptors):
                    accepted_ballot, _ = state.get(f'accepted_{j}', (0, None))
                    if accepted_ballot == ballot:
                        promises += 1

                if promises > self.num_acceptors // 2:
                    # Proposer can propose a value
                    for value in ['A', 'B']:  # Possible values
                        new_state = state.copy()
                        new_state[f'proposed_{i}'] = value
                        next_states.append(new_state)

        # Phase 2b: Acceptor accepts
        for i in range(self.num_acceptors):
            current_ballot, current_value = state.get(f'accepted_{i}', (0, None))

            # Find proposers with matching ballot
            for j in range(self.num_proposers):
                ballot = state.get(f'ballot_{j}', 0)
                value = state.get(f'proposed_{j}')

                if value is not None and ballot == current_ballot:
                    new_state = state.copy()
                    new_state[f'accepted_{i}'] = (ballot, value)
                    next_states.append(new_state)

        # Learn decision
        # Check if any value has majority acceptance
        for value in ['A', 'B']:
            accepts = 0
            for i in range(self.num_acceptors):
                _, accepted_value = state.get(f'accepted_{i}', (0, None))
                if accepted_value == value:
                    accepts += 1

            if accepts > self.num_acceptors // 2 and state.get('decision') is None:
                new_state = state.copy()
                new_state['decision'] = value
                next_states.append(new_state)

        return next_states

    def verify_safety(self) -> Dict[str, Any]:
        """
        Verify Paxos safety properties
        """
        results = {}

        # Build state machine
        sm = StateMachine(
            states=self.states,
            initial_state=self._state_to_str({
                f'ballot_{i}': 0 for i in range(self.num_proposers)
            }),
            transitions=dict(self.transitions)
        )

        # Add propositions
        for state_str in self.states:
            # Parse state string
            parts = state_str.split('|')
            decision = None
            for part in parts:
                if part.startswith('D='):
                    decision = part[2:] if part[2:] != 'None' else None

            if decision:
                sm.add_proposition(state_str, f"decided_{decision}")

            # Check for conflicting decisions (safety violation)
            # In real implementation, we'd check more properties

        # Check Agreement: Only one value can be decided
        agreement_formula = LTLFormula("implies",
            LTLFormula("and",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")),
            LTLFormula("atomic", "false"))  # A and B cannot both be true

        satisfied, counterexample = sm.check_invariant(agreement_formula)
        results['agreement'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        # Check Validity: Only proposed values can be decided
        # Simplified check
        validity_formula = LTLFormula("implies",
            LTLFormula("or",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")),
            LTLFormula("atomic", "value_proposed"))

        satisfied, counterexample = sm.check_invariant(validity_formula)
        results['validity'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        # Check Termination: Eventually a decision is reached
        termination_formula = LTLFormula("eventually",
            LTLFormula("or",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")))

        satisfied, counterexample = sm.check_liveness(termination_formula)
        results['termination'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        return results

class SPINModelChecker:
    """
    Simplified SPIN-like model checker with Promela-like syntax
    """

    def __init__(self):
        self.processes = []
        self.channels = {}
        self.variables = {}
        self.ltl_properties = []

    def add_process(self, name: str, code: str):
        """Add a process with Promela-like code"""
        self.processes.append({
            'name': name,
            'code': code,
            'pc': 0,  # Program counter
            'local_vars': {}
        })

    def add_channel(self, name: str, capacity: int):
        """Add a message channel"""
        self.channels[name] = {
            'queue': [],
            'capacity': capacity
        }

    def add_variable(self, name: str, initial_value):
        """Add global variable"""
        self.variables[name] = initial_value

    def add_ltl_property(self, formula: str):
        """Add LTL property to verify"""
        self.ltl_properties.append(formula)

    def parse_promela(self, code: str):
        """Parse simplified Promela code"""
        lines = [line.strip() for line in code.split('\n') if line.strip()]

        for line in lines:
            if line.startswith('chan '):
                # Channel declaration
                parts = line.split()
                name = parts[1]
                capacity = int(parts[3]) if len(parts) > 3 else 0
                self.add_channel(name, capacity)

            elif line.startswith('int '):
                # Variable declaration
                parts = line.split()
                name = parts[1]
                value = int(parts[3]) if len(parts) > 3 else 0
                self.add_variable(name, value)

            elif line.startswith('proctype '):
                # Process type
                name = line.split()[1].rstrip('()')
                # Simplified - would need to parse body

    def model_check(self, max_steps: int = 1000) -> Dict[str, Any]:
        """
        Perform model checking with state space exploration
        """
        # Initial state
        initial_state = {
            'processes': [p.copy() for p in self.processes],
            'channels': {k: v.copy() for k, v in self.channels.items()},
            'variables': self.variables.copy(),
            'step': 0
        }

        visited = set()
        queue = [(initial_state, [])]  # (state, trace)
        violations = []

        while queue:
            state, trace = queue.pop(0)

            # Check for LTL violations in current trace
            for formula in self.ltl_properties:
                if not self._check_ltl_trace(formula, trace + [state]):
                    violations.append({
                        'formula': formula,
                        'trace': trace + [state]
                    })

            # Check if we should stop
            if len(trace) >= max_steps:
                continue

            # Generate next states
            next_states = self._get_next_states(state)

            for next_state in next_states:
                state_hash = self._hash_state(next_state)
                if state_hash not in visited:
                    visited.add(state_hash)
                    queue.append((next_state, trace + [state]))

        return {
            'states_explored': len(visited),
            'violations': violations,
            'safe': len(violations) == 0
        }

    def _hash_state(self, state: Dict) -> str:
        """Create hash for state to detect duplicates"""
        # Simplified - would need proper hashing
        return str(state)

    def _get_next_states(self, state: Dict) -> List[Dict]:
        """Generate all possible next states"""
        # This is a simplified implementation
        # Real implementation would parse Promela code and execute steps
        next_states = []

        # Simulate some transitions
        # Send/receive on channels, variable updates, etc.

        return next_states

    def _check_ltl_trace(self, formula: str, trace: List[Dict]) -> bool:
        """Check LTL formula on trace"""
        # Simplified implementation
        # Real implementation would parse LTL and evaluate
        return True

def demonstrate_formal_verification():
    """
    Complete demonstration of formal verification techniques
    """
    print("="*70)
    print("FORMAL VERIFICATION OF DISTRIBUTED PROTOCOLS")
    print("="*70)

    # 1. LTL Formula Examples
    print("\n1. Linear Temporal Logic (LTL) Examples:")
    print("-"*40)

    # Define atomic propositions
    p = LTLFormula("atomic", "connected")
    q = LTLFormula("atomic", "data_sent")
    r = LTLFormula("atomic", "ack_received")

    # Safety: Always, if connected then eventually data sent
    safety = LTLFormula("always", 
        LTLFormula("implies", p, LTLFormula("eventually", q)))

    # Liveness: Eventually ack received
    liveness = LTLFormula("eventually", r)

    # Response: Always, if data sent then eventually ack received
    response = LTLFormula("always",
        LTLFormula("implies", q, LTLFormula("eventually", r)))

    print(f"Safety formula: {safety.to_str()}")
    print(f"Liveness formula: {liveness.to_str()}")
    print(f"Response formula: {response.to_str()}")

    # Test evaluation
    trace = [
        {"connected"},
        {"connected", "data_sent"},
        {"connected", "ack_received"},
        {"ack_received"}
    ]

    print(f"\nTrace: {trace}")
    print(f"Safety holds: {safety.evaluate(trace)}")
    print(f"Liveness holds: {liveness.evaluate(trace)}")
    print(f"Response holds: {response.evaluate(trace)}")

    # 2. State Machine Model Checking
    print("\n\n2. State Machine Model Checking:")
    print("-"*40)

    # Create a simple state machine
    states = {"S0", "S1", "S2", "S3"}
    transitions = {
        "S0": ["S1", "S2"],
        "S1": ["S3"],
        "S2": ["S3"],
        "S3": []
    }

    sm = StateMachine(states, "S0", transitions)

    # Add propositions
    sm.add_proposition("S0", "initial")
    sm.add_proposition("S1", "processing")
    sm.add_proposition("S2", "processing")
    sm.add_proposition("S3", "finished")

    # Check invariant: always (processing -> not finished)
    invariant = LTLFormula("implies",
        LTLFormula("atomic", "processing"),
        LTLFormula("not", LTLFormula("atomic", "finished")))

    satisfied, counterexample = sm.check_invariant(invariant)
    print(f"Invariant '{invariant.to_str()}' satisfied: {satisfied}")
    if counterexample:
        print(f"Counterexample: {counterexample}")

    # Check liveness: eventually finished
    liveness_prop = LTLFormula("eventually", LTLFormula("atomic", "finished"))
    satisfied, counterexample = sm.check_liveness(liveness_prop)
    print(f"\nLiveness '{liveness_prop.to_str()}' satisfied: {satisfied}")

    # 3. Paxos Model Checking
    print("\n\n3. Paxos Consensus Protocol Verification:")
    print("-"*40)

    checker = PaxosModelChecker(num_proposers=2, num_acceptors=3)
    results = checker.verify_safety()

    print("Paxos Safety Properties:")
    for prop, result in results.items():
        print(f"  {prop}: {'✓' if result['satisfied'] else '✗'}")
        if not result['satisfied'] and result['counterexample']:
            print(f"    Counterexample length: {len(result['counterexample'])}")

    # 4. Promela-like Model Checking
    print("\n\n4. Promela/SPIN-style Model Checking:")
    print("-"*40)

    spin = SPINModelChecker()

    # Define a simple mutual exclusion protocol
    promela_code = """
    bool turn = 0;
    bool flag[2] = 0;

    proctype P(int id) {
        flag[id] = 1;
        turn = 1 - id;
        (flag[1-id] == 0 || turn == id);
        /* critical section */
        flag[id] = 0;
    }
    """

    spin.parse_promela(promela_code)

    # Add LTL properties
    spin.add_ltl_property("[] (P0_in_critical -> !P1_in_critical)")  # Mutual exclusion
    spin.add_ltl_property("[] (P0_wants -> <> P0_in_critical)")  # Liveness

    print("Mutual exclusion protocol defined")
    print("LTL properties added for verification")

    # 5. Theorem Proving Concepts
    print("\n\n5. Theorem Proving Concepts:")
    print("-"*40)

    print("Hoare Logic for Protocol Verification:")
    print("  {P} S {Q} - If precondition P holds before S, then Q holds after")
    print("\nExample: Two-Phase Commit")
    print("  Precondition: All participants in init state")
    print("  Protocol: Coordinator sends prepare -> participants vote -> decide")
    print("  Postcondition: All participants have same decision (commit/abort)")

    print("\nInvariant Proof Technique:")
    print("  1. Base case: Invariant holds in initial state")
    print("  2. Inductive step: If invariant holds and action occurs,")
    print("     invariant still holds in next state")
    print("  3. Conclusion: Invariant holds for all reachable states")

def verify_distributed_mutex():
    """
    Complete verification of distributed mutual exclusion protocol
    """
    print("\n" + "="*70)
    print("COMPLETE VERIFICATION: DISTRIBUTED MUTUAL EXCLUSION")
    print("="*70)

    # Define the protocol states
    class MutexState(Enum):
        RELEASED = "released"
        WANTED = "wanted"
        HELD = "held"

    # Create state machine for Ricart-Agrawala algorithm
    states = set()
    transitions = defaultdict(list)

    # Generate all possible states (simplified)
    # 2 processes, each can be in 3 states, plus timestamp state
    for p1_state in MutexState:
        for p2_state in MutexState:
            for ts1 in [0, 1, 2]:
                for ts2 in [0, 1, 2]:
                    state_str = f"P1:{p1_state.value}:{ts1}|P2:{p2_state.value}:{ts2}"
                    states.add(state_str)

    # Define transitions based on Ricart-Agrawala algorithm
    # Simplified version
    for state_str in states:
        parts = state_str.split('|')
        p1_part, p2_part = parts

        p1_state, p1_ts = p1_part.split(':')[1], int(p1_part.split(':')[2])
        p2_state, p2_ts = p2_part.split(':')[1], int(p2_part.split(':')[2])

        # Process 1 can request critical section
        if p1_state == MutexState.RELEASED.value:
            new_state = f"P1:{MutexState.WANTED.value}:{max(p1_ts, p2_ts) + 1}|{p2_part}"
            transitions[state_str].append(new_state)

        # Process 1 can enter critical section if it has smallest timestamp
        if (p1_state == MutexState.WANTED.value and 
            (p2_state != MutexState.WANTED.value or p1_ts < p2_ts or 
             (p1_ts == p2_ts and 1 < 2))):  # Process IDs break ties
            new_state = f"P1:{MutexState.HELD.value}:{p1_ts}|{p2_part}"
            transitions[state_str].append(new_state)

        # Process 1 can release
        if p1_state == MutexState.HELD.value:
            new_state = f"P1:{MutexState.RELEASED.value}:{p1_ts}|{p2_part}"
            transitions[state_str].append(new_state)

        # Similar transitions for process 2
        if p2_state == MutexState.RELEASED.value:
            new_state = f"{p1_part}|P2:{MutexState.WANTED.value}:{max(p1_ts, p2_ts) + 1}"
            transitions[state_str].append(new_state)

        if (p2_state == MutexState.WANTED.value and 
            (p1_state != MutexState.WANTED.value or p2_ts < p1_ts or 
             (p2_ts == p1_ts and 2 < 1))):
            new_state = f"{p1_part}|P2:{MutexState.HELD.value}:{p2_ts}"
            transitions[state_str].append(new_state)

        if p2_state == MutexState.HELD.value:
            new_state = f"{p1_part}|P2:{MutexState.RELEASED.value}:{p2_ts}"
            transitions[state_str].append(new_state)

    # Create state machine
    initial_state = f"P1:{MutexState.RELEASED.value}:0|P2:{MutexState.RELEASED.value}:0"
    sm = StateMachine(states, initial_state, dict(transitions))

    # Add propositions
    for state_str in states:
        if "HELD" in state_str:
            if "P1:HELD" in state_str:
                sm.add_proposition(state_str, "P1_in_critical")
            if "P2:HELD" in state_str:
                sm.add_proposition(state_str, "P2_in_critical")

        if "WANTED" in state_str:
            if "P1:WANTED" in state_str:
                sm.add_proposition(state_str, "P1_wants")
            if "P2:WANTED" in state_str:
                sm.add_proposition(state_str, "P2_wants")

    # Verify properties
    print("\nVerifying Mutual Exclusion Properties:")

    # 1. Mutual exclusion: Never both processes in critical section
    mutex_formula = LTLFormula("always",
        LTLFormula("not",
            LTLFormula("and",
                LTLFormula("atomic", "P1_in_critical"),
                LTLFormula("atomic", "P2_in_critical"))))

    satisfied, counterexample = sm.check_invariant(mutex_formula)
    print(f"  1. Mutual Exclusion: {'✓' if satisfied else '✗'}")
    if not satisfied and counterexample:
        print(f"     Violation found in trace of length {len(counterexample)}")

    # 2. Deadlock freedom: If process wants CS, eventually gets it
    deadlock_free = LTLFormula("always",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_wants"),
            LTLFormula("eventually", LTLFormula("atomic", "P1_in_critical"))))

    satisfied, counterexample = sm.check_invariant(deadlock_free)
    print(f"  2. Deadlock Freedom: {'✓' if satisfied else '✗'}")

    # 3. Starvation freedom: Eventually every request is granted
    # Simplified check
    no_starvation = LTLFormula("eventually",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_wants"),
            LTLFormula("atomic", "P1_in_critical")))

    satisfied, counterexample = sm.check_liveness(no_starvation)
    print(f"  3. No Starvation: {'✓' if satisfied else '✗'}")

    # 4. Safety: Process only enters CS when it has permission
    # Based on timestamp comparison
    safety = LTLFormula("always",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_in_critical"),
            LTLFormula("or",
                LTLFormula("not", LTLFormula("atomic", "P2_wants")),
                LTLFormula("atomic", "P1_has_smaller_timestamp"))))  # Simplified

    satisfied, counterexample = sm.check_invariant(safety)
    print(f"  4. Safety (timestamp order): {'✓' if satisfied else '✗'}")

    # Generate some example traces
    print("\nExample Protocol Execution Traces:")
    traces = sm.get_all_traces(max_length=5)

    for i, trace in enumerate(traces[:3]):  # Show first 3
        print(f"\n  Trace {i+1}:")
        for j, state in enumerate(trace):
            print(f"    Step {j}: {state}")

def mathematical_proofs_formal_verification():
    """
    Mathematical proofs for formal verification
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF FORMAL VERIFICATION")
    print("="*70)

    print("\nTheorem 1: Soundness of Hoare Logic")
    print("Proof:")
    print("  Let {P} S {Q} be a Hoare triple.")
    print("  By induction on the structure of S:")
    print("    Base case: Skip: {P} skip {P} (trivially true)")
    print("    Assignment: {P[e/x]} x := e {P}")
    print("    Sequence: {P} S1; S2 {R} if {P} S1 {Q} and {Q} S2 {R}")
    print("    If: {P ∧ B} S1 {Q} and {P ∧ ¬B} S2 {Q} ⇒ {P} if B then S1 else S2 {Q}")
    print("    While: {P ∧ B} S {P} ⇒ {P} while B do S {P ∧ ¬B}")
    print("  Each rule preserves truth. ✓")

    print("\nTheorem 2: Completeness of Hoare Logic (Cook's Theorem)")
    print("Proof (sketch):")
    print("  1. For any program S and assertions P, Q,")
    print("     if {P} S {Q} is true in all interpretations,")
    print("     then it is provable in Hoare logic.")
    print("  2. Construct weakest precondition wp(S, Q)")
    print("  3. Show P → wp(S, Q) is valid")
    print("  4. By Gödel's completeness theorem, valid formulas are provable.")
    print("  5. Therefore, {P} S {Q} is provable. ✓")

    print("\nTheorem 3: Floyd-Hoare Method for Loop Invariants")
    print("Proof:")
    print("  For loop: while B do S")
    print("  Find invariant I such that:")
    print("    1. P → I  (initialization)")
    print("    2. {I ∧ B} S {I}  (preservation)")
    print("    3. I ∧ ¬B → Q  (postcondition)")
    print("  Then by while rule, {P} while B do S {Q}")
    print("  This is complete for loops with well-founded ordering. ✓")

    print("\nTheorem 4: Temporal Logic Model Checking (CTL*)")
    print("Proof (complexity):")
    print("  CTL model checking: O(|S| × |φ|) where S is state space")
    print("  LTL model checking: PSPACE-complete in formula size")
    print("  CTL* model checking: EXPTIME-complete")
    print("  Proof via reduction from QBF (Quantified Boolean Formulas). ✓")

if __name__ == "__main__":
    demonstrate_formal_verification()
    verify_distributed_mutex()
    mathematical_proofs_formal_verification()

8. Zero-Knowledge Proofs in Distributed Systems

Mathematical Foundations of ZK Proofs

Zero-Knowledge Proofs allow proving a statement without revealing why it's true:

Three Properties:

Completeness: If statement is true, honest verifier will be convinced
Soundness: If statement is false, no cheating prover can convince verifier
Zero-knowledge: Verifier learns nothing beyond statement's truth

Mathematical Framework:

Σ-protocols: 3-move protocols (commit, challenge, response)
Fiat-Shamir Heuristic: Convert interactive to non-interactive using hash functions
zk-SNARKs: Succinct Non-interactive ARguments of Knowledge
- Uses quadratic arithmetic programs and pairing-based cryptography

Complete Python Implementation: zk-SNARKs and Bulletproofs

import hashlib
import random
from typing import List, Tuple, Dict, Any, Optional
from dataclasses import dataclass
from enum import Enum
import math
import json
from functools import lru_cache
import time

# Elliptic curve parameters (simplified for educational purposes)
class EllipticCurve:
    """
    Simplified elliptic curve implementation for ZK proofs
    Using secp256k1 parameters (like Bitcoin)
    """

    def __init__(self):
        # secp256k1 parameters
        self.p = 2**256 - 2**32 - 977  # Field prime
        self.a = 0
        self.b = 7
        self.Gx = 0x79BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798
        self.Gy = 0x483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8
        self.n = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141  # Order

        self.G = (self.Gx, self.Gy)

    def point_add(self, P: Tuple[int, int], Q: Tuple[int, int]) -> Tuple[int, int]:
        """Elliptic curve point addition"""
        if P == (0, 0):
            return Q
        if Q == (0, 0):
            return P

        x1, y1 = P
        x2, y2 = Q

        if x1 == x2 and y1 != y2:
            return (0, 0)  # Point at infinity

        if P == Q:
            # Point doubling
            lam = (3 * x1 * x1 + self.a) * pow(2 * y1, -1, self.p) % self.p
        else:
            lam = (y2 - y1) * pow(x2 - x1, -1, self.p) % self.p

        x3 = (lam * lam - x1 - x2) % self.p
        y3 = (lam * (x1 - x3) - y1) % self.p

        return (x3, y3)

    def scalar_mult(self, k: int, P: Tuple[int, int]) -> Tuple[int, int]:
        """Scalar multiplication k * P"""
        result = (0, 0)  # Point at infinity
        addend = P

        while k > 0:
            if k & 1:
                result = self.point_add(result, addend)
            addend = self.point_add(addend, addend)
            k >>= 1

        return result

    def hash_to_point(self, data: bytes) -> Tuple[int, int]:
        """Hash data to point on curve"""
        # Simplified: use hash mod p as x, compute y
        h = int.from_bytes(hashlib.sha256(data).digest(), 'big') % self.p

        # Find y such that y² = x³ + ax + b
        x = h
        rhs = (x * x * x + self.a * x + self.b) % self.p

        # Try to find square root (simplified)
        y = pow(rhs, (self.p + 1) // 4, self.p)  # Works for p ≡ 3 mod 4

        return (x, y)

class SigmaProtocol:
    """
    Σ-protocol for discrete logarithm knowledge
    Proves knowledge of x such that y = g^x
    """

    def __init__(self, curve: EllipticCurve):
        self.curve = curve

    def prove(self, x: int, y: Tuple[int, int]) -> Tuple[Tuple[int, int], int, int]:
        """
        Generate proof of knowledge of discrete logarithm

        Returns: (commitment, challenge, response)
        """
        # Step 1: Prover chooses random r
        r = random.randint(1, self.curve.n - 1)

        # Step 2: Compute commitment t = g^r
        t = self.curve.scalar_mult(r, self.curve.G)

        # Step 3: Verifier sends random challenge c
        # In interactive protocol, verifier sends c
        # For non-interactive, we generate using Fiat-Shamir
        c = self._generate_challenge(y, t)

        # Step 4: Prover computes response s = r + c * x mod n
        s = (r + c * x) % self.curve.n

        return (t, c, s)

    def verify(self, y: Tuple[int, int], proof: Tuple[Tuple[int, int], int, int]) -> bool:
        """
        Verify proof of knowledge
        """
        t, c, s = proof

        # Check: g^s == t * y^c
        left = self.curve.scalar_mult(s, self.curve.G)
        right = self.curve.point_add(t, self.curve.scalar_mult(c, y))

        return left == right

    def _generate_challenge(self, y: Tuple[int, int], t: Tuple[int, int]) -> int:
        """Fiat-Shamir heuristic to make non-interactive"""
        data = json.dumps({
            'y': y,
            't': t
        }).encode()

        # Hash to get challenge
        h = hashlib.sha256(data).digest()
        return int.from_bytes(h, 'big') % self.curve.n

class Bulletproofs:
    """
    Bulletproofs implementation for range proofs
    Proves a committed value is within [0, 2^n - 1]
    """

    def __init__(self, curve: EllipticCurve, n_bits: int = 32):
        self.curve = curve
        self.n_bits = n_bits

        # Generators for Pedersen commitments
        self.G = curve.G
        self.H = curve.hash_to_point(b"bulletproofs_H_generator")

    def commit(self, value: int, blinding: int) -> Tuple[int, int]:
        """
        Pedersen commitment: C = v*G + r*H
        """
        vG = self.curve.scalar_mult(value, self.G)
        rH = self.curve.scalar_mult(blinding, self.H)
        return self.curve.point_add(vG, rH)

    def prove_range(self, value: int, blinding: int, 
                   commitment: Tuple[int, int]) -> Dict[str, Any]:
        """
        Generate bulletproof range proof
        """
        # Decompose value into bits
        bits = [(value >> i) & 1 for i in range(self.n_bits)]

        # Commit to each bit
        aL = bits  # Left vector
        aR = [b ^ 1 for b in bits]  # Right vector: aR = aL - 1

        # Generate random blinding factors
        alpha = random.randint(1, self.curve.n - 1)
        sL = [random.randint(1, self.curve.n - 1) for _ in range(self.n_bits)]
        sR = [random.randint(1, self.curve.n - 1) for _ in range(self.n_bits)]
        rho = random.randint(1, self.curve.n - 1)

        # Compute vector commitments
        A = self._vector_commit(aL, aR, alpha)
        S = self._vector_commit(sL, sR, rho)

        # Fiat-Shamir challenge
        y = self._hash_challenge(commitment, A, S)
        z = self._hash_challenge(commitment, A, S, y)

        # Compute polynomials
        l0, l1, r0, r1 = self._compute_polynomials(aL, aR, sL, sR, y, z)

        # Inner product argument
        t1, t2, tau1, tau2 = self._inner_product_argument(l0, l1, r0, r1)

        # Final challenge
        x = self._hash_challenge(commitment, A, S, y, z, t1, t2)

        # Compute responses
        taux = (tau1 * x + tau2 * x * x) % self.curve.n
        mu = (alpha + rho * x) % self.curve.n

        # Compute inner product
        inner_product = sum(l * r for l, r in zip(l0, r0))

        return {
            'A': A,
            'S': S,
            't1': t1,
            't2': t2,
            'taux': taux,
            'mu': mu,
            'inner_product': inner_product,
            'commitment': commitment,
            'L': [],  # Would contain intermediate L values
            'R': [],  # Would contain intermediate R values
            'a': 0,   # Final a
            'b': 0    # Final b
        }

    def verify_range(self, proof: Dict[str, Any]) -> bool:
        """
        Verify bulletproof range proof
        """
        # This is a simplified verification
        # Real bulletproofs have more complex verification

        commitment = proof['commitment']

        # Recompute challenges
        y = self._hash_challenge(commitment, proof['A'], proof['S'])
        z = self._hash_challenge(commitment, proof['A'], proof['S'], y)
        x = self._hash_challenge(commitment, proof['A'], proof['S'], y, z, 
                                proof['t1'], proof['t2'])

        # Check commitment reconstruction
        # In real implementation, would verify inner product argument

        # Simplified check
        return proof['inner_product'] < 2**self.n_bits

    def _vector_commit(self, a: List[int], b: List[int], alpha: int) -> Tuple[int, int]:
        """Commit to two vectors"""
        # Simplified: C = Σ(a_i * G_i) + Σ(b_i * H_i) + alpha * H
        commitment = (0, 0)

        for i in range(self.n_bits):
            # Use different generators for each position
            Gi = self.curve.hash_to_point(f"G_{i}".encode())
            Hi = self.curve.hash_to_point(f"H_{i}".encode())

            aGi = self.curve.scalar_mult(a[i], Gi)
            bHi = self.curve.scalar_mult(b[i], Hi)

            commitment = self.curve.point_add(commitment, aGi)
            commitment = self.curve.point_add(commitment, bHi)

        # Add blinding
        alphaH = self.curve.scalar_mult(alpha, self.H)
        commitment = self.curve.point_add(commitment, alphaH)

        return commitment

    def _compute_polynomials(self, aL, aR, sL, sR, y, z):
        """Compute polynomial coefficients"""
        # Compute l(x) = aL - z*1^n + sL*x
        # Compute r(x) = y^n ∘ (aR + z*1^n + sR*x) + z^2 * 2^n

        y_vector = [pow(y, i, self.curve.n) for i in range(self.n_bits)]
        two_vector = [pow(2, i, self.curve.n) for i in range(self.n_bits)]

        # l0 = aL - z*1^n
        l0 = [(aL[i] - z) % self.curve.n for i in range(self.n_bits)]
        l1 = sL  # Coefficient for x

        # r0 = y^n ∘ (aR + z*1^n) + z^2 * 2^n
        r0 = []
        for i in range(self.n_bits):
            term1 = (aR[i] + z) % self.curve.n
            term2 = (z * z * two_vector[i]) % self.curve.n
            r0.append((y_vector[i] * term1 + term2) % self.curve.n)

        # r1 = y^n ∘ sR
        r1 = [(y_vector[i] * sR[i]) % self.curve.n for i in range(self.n_bits)]

        return l0, l1, r0, r1

    def _inner_product_argument(self, l0, l1, r0, r1):
        """Generate inner product argument"""
        # Simplified - real implementation uses recursive protocol

        # Compute t1 = <l1, r0> + <l0, r1>
        t1 = sum(l1[i] * r0[i] + l0[i] * r1[i] for i in range(self.n_bits)) % self.curve.n

        # Compute t2 = <l1, r1>
        t2 = sum(l1[i] * r1[i] for i in range(self.n_bits)) % self.curve.n

        # Random blinding factors
        tau1 = random.randint(1, self.curve.n - 1)
        tau2 = random.randint(1, self.curve.n - 1)

        return t1, t2, tau1, tau2

    def _hash_challenge(self, *args) -> int:
        """Generate Fiat-Shamir challenge"""
        data = json.dumps(args, default=str).encode()
        h = hashlib.sha256(data).digest()
        return int.from_bytes(h, 'big') % self.curve.n

class zkSNARK:
    """
    Simplified zk-SNARK implementation
    Proves correct execution of arithmetic circuit
    """

    def __init__(self, curve: EllipticCurve):
        self.curve = curve

        # Setup parameters
        self.tau = random.randint(1, self.curve.n - 1)  # Toxic waste (should be destroyed)

        # CRS (Common Reference String)
        self.g1 = curve.G
        self.g2 = curve.hash_to_point(b"zksnark_g2")

        # Powers of tau in G1 and G2
        self.powers_g1 = [curve.scalar_mult(pow(self.tau, i, curve.n), self.g1) 
                         for i in range(10)]  # Up to degree 9
        self.powers_g2 = [curve.scalar_mult(pow(self.tau, i, curve.n), self.g2) 
                         for i in range(10)]

    def setup(self, circuit):
        """
        Generate proving and verification keys for circuit
        """
        # QAP (Quadratic Arithmetic Program) setup
        # Simplified - real implementation uses polynomial interpolation

        # For circuit: out = in1 * in2 (multiplication gate)
        # Represent as: out - in1 * in2 = 0

        proving_key = {
            'g1': self.g1,
            'g2': self.g2,
            'alpha_g1': self.curve.scalar_mult(self.tau, self.g1),  # αG1
            'beta_g1': self.curve.scalar_mult(self.tau * 2, self.g1),  # βG1
            'beta_g2': self.curve.scalar_mult(self.tau * 2, self.g2),  # βG2
            'powers_g1': self.powers_g1,
            'powers_g2': self.powers_g2,
            'gamma_inv': pow(self.tau * 3, -1, self.curve.n),  # γ^{-1}
            'delta_inv': pow(self.tau * 4, -1, self.curve.n),  # δ^{-1}
        }

        verification_key = {
            'g1': self.g1,
            'g2': self.g2,
            'alpha_g1_beta_g2': self.curve.point_add(
                self.curve.scalar_mult(self.tau, self.g1),
                self.curve.scalar_mult(self.tau * 2, self.g2)
            ),
            'gamma_g2': self.curve.scalar_mult(self.tau * 3, self.g2),
            'delta_g2': self.curve.scalar_mult(self.tau * 4, self.g2),
            'ic': [  # Input consistency
                self.curve.scalar_mult(pow(self.tau, i, self.curve.n), self.g1)
                for i in range(3)  # For 2 inputs + output
            ]
        }

        return proving_key, verification_key

    def prove(self, proving_key, inputs, output):
        """
        Generate proof for circuit execution
        """
        # Example circuit: out = in1 * in2

        # Compute witness polynomial
        # In real zk-SNARK, this involves:
        # 1. Computing witness (assignment to wires)
        # 2. Computing polynomial coefficients
        # 3. Evaluating at tau

        # Simplified proof generation
        proof = {
            'a_g1': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g1']),
            'b_g2': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g2']),
            'c_g1': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g1']),
            'inputs': inputs,
            'output': output
        }

        return proof

    def verify(self, verification_key, proof, public_inputs):
        """
        Verify zk-SNARK proof
        """
        # Simplified verification
        # Real verification uses pairing checks:
        # e(A, B) = e(α, β) * e(C, δ) ...

        # Check input consistency
        ic_sum = (0, 0)
        for i, inp in enumerate(public_inputs):
            ic_term = self.curve.scalar_mult(inp, verification_key['ic'][i])
            ic_sum = self.curve.point_add(ic_sum, ic_term)

        # Check circuit correctness (simplified)
        # For multiplication gate: out = in1 * in2
        if len(public_inputs) >= 2:
            in1, in2 = public_inputs[:2]
            out = proof['output']

            if out != in1 * in2:
                return False

        return True

class DistributedZK:
    """
    Distributed zero-knowledge proofs for distributed systems
    """

    def __init__(self):
        self.curve = EllipticCurve()
        self.sigma = SigmaProtocol(self.curve)
        self.bulletproofs = Bulletproofs(self.curve)
        self.zksnark = zkSNARK(self.curve)

    def prove_membership(self, secret: int, public_set: List[Tuple[int, int]]) -> Dict[str, Any]:
        """
        Prove membership in set without revealing which element
        """
        # Each element: y_i = g^x_i
        # Prover knows x such that y = g^x is in set

        # Commit to random values
        r = random.randint(1, self.curve.n - 1)
        C = self.curve.scalar_mult(r, self.curve.G)

        # For each element, create proof OR proof
        proofs = []
        for y in public_set:
            # Create simulated proof for elements we don't know
            # and real proof for the one we know

            # This is simplified - real implementation uses OR composition
            if self.curve.scalar_mult(secret, self.curve.G) == y:
                # Real proof
                proof = self.sigma.prove(secret, y)
            else:
                # Simulated proof
                proof = self._simulate_proof(y)

            proofs.append(proof)

        return {
            'commitment': C,
            'proofs': proofs,
            'set_hash': hashlib.sha256(
                str(sorted(public_set)).encode()
            ).hexdigest()
        }

    def _simulate_proof(self, y: Tuple[int, int]) -> Tuple[Tuple[int, int], int, int]:
        """Simulate proof for element we don't know"""
        # Choose random challenge and response
        c = random.randint(1, self.curve.n - 1)
        s = random.randint(1, self.curve.n - 1)

        # Compute commitment that matches: t = g^s * y^{-c}
        sG = self.curve.scalar_mult(s, self.curve.G)
        neg_cY = self.curve.scalar_mult(-c % self.curve.n, y)
        t = self.curve.point_add(sG, neg_cY)

        return (t, c, s)

    def verify_membership(self, proof: Dict[str, Any], public_set: List[Tuple[int, int]]) -> bool:
        """
        Verify membership proof
        """
        # Check all proofs
        for i, (y, proof_i) in enumerate(zip(public_set, proof['proofs'])):
            if not self.sigma.verify(y, proof_i):
                return False

        return True

    def prove_shuffle(self, input_list: List[Tuple[int, int]], 
                     output_list: List[Tuple[int, int]], 
                     permutation: List[int]) -> Dict[str, Any]:
        """
        Prove shuffle (mixnet) without revealing permutation
        """
        # Shuffle proof using Neff's technique

        # Commit to permutation matrix
        n = len(input_list)
        r = random.randint(1, self.curve.n - 1)

        # Create commitments to permutation
        commitments = []
        for i in range(n):
            # Commit to each row of permutation matrix
            C = self.curve.scalar_mult(r + i, self.curve.G)
            commitments.append(C)

        # Prove that output is permutation of input
        # Simplified - real implementation uses complex proofs

        return {
            'input_hash': hashlib.sha256(str(input_list).encode()).hexdigest(),
            'output_hash': hashlib.sha256(str(output_list).encode()).hexdigest(),
            'commitments': commitments,
            'proof': 'simplified_shuffle_proof'
        }

    def verify_shuffle(self, proof: Dict[str, Any], 
                      input_list: List[Tuple[int, int]], 
                      output_list: List[Tuple[int, int]]) -> bool:
        """
        Verify shuffle proof
        """
        # Check hashes
        input_hash = hashlib.sha256(str(input_list).encode()).hexdigest()
        output_hash = hashlib.sha256(str(output_list).encode()).hexdigest()

        if input_hash != proof['input_hash']:
            return False

        if output_hash != proof['output_hash']:
            return False

        # Simplified verification
        # Real verification would check polynomial equations

        return True

def demonstrate_zk_proofs():
    """
    Complete demonstration of zero-knowledge proofs
    """
    print("="*70)
    print("ZERO-KNOWLEDGE PROOFS IN DISTRIBUTED SYSTEMS")
    print("="*70)

    dist_zk = DistributedZK()

    # 1. Sigma Protocol (Discrete Log Proof)
    print("\n1. Σ-Protocol for Discrete Logarithm:")
    print("-"*40)

    # Prover knows x such that y = g^x
    x = random.randint(1, dist_zk.curve.n - 1)
    y = dist_zk.curve.scalar_mult(x, dist_zk.curve.G)

    print(f"Secret x: {x}")
    print(f"Public y: {y}")

    # Generate proof
    proof = dist_zk.sigma.prove(x, y)
    print(f"\nProof generated: (t, c, s)")
    print(f"  t (commitment): {proof[0]}")
    print(f"  c (challenge): {proof[1]}")
    print(f"  s (response): {proof[2]}")

    # Verify proof
    valid = dist_zk.sigma.verify(y, proof)
    print(f"\nProof valid: {'✓' if valid else '✗'}")

    # 2. Bulletproofs (Range Proofs)
    print("\n\n2. Bulletproofs for Range Proofs:")
    print("-"*40)

    value = 42
    blinding = random.randint(1, dist_zk.curve.n - 1)
    commitment = dist_zk.bulletproofs.commit(value, blinding)

    print(f"Value: {value} (0 ≤ value < {2**32})")
    print(f"Commitment: {commitment}")

    # Generate range proof
    range_proof = dist_zk.bulletproofs.prove_range(value, blinding, commitment)
    print(f"\nRange proof generated")
    print(f"  Proof size: ~{len(str(range_proof))} characters")

    # Verify range proof
    valid = dist_zk.bulletproofs.verify_range(range_proof)
    print(f"Range proof valid: {'✓' if valid else '✗'}")

    # 3. Set Membership Proof
    print("\n\n3. Set Membership Proof:")
    print("-"*40)

    # Create public set
    public_set = []
    secrets = []

    for i in range(5):
        secret = random.randint(1, dist_zk.curve.n - 1)
        point = dist_zk.curve.scalar_mult(secret, dist_zk.curve.G)
        public_set.append(point)
        secrets.append(secret)

    # Prover knows one secret
    known_secret = secrets[2]  # Know 3rd element
    print(f"Public set has {len(public_set)} elements")
    print(f"Prover knows secret for element 3")

    # Generate membership proof
    membership_proof = dist_zk.prove_membership(known_secret, public_set)
    print(f"\nMembership proof generated")
    print(f"  Set hash: {membership_proof['set_hash'][:16]}...")

    # Verify membership proof
    valid = dist_zk.verify_membership(membership_proof, public_set)
    print(f"Membership proof valid: {'✓' if valid else '✗'}")

    # 4. Shuffle Proof (Mixnet)
    print("\n\n4. Shuffle Proof (Mixnet):")
    print("-"*40)

    # Create input list
    input_list = []
    for i in range(3):
        r = random.randint(1, dist_zk.curve.n - 1)
        point = dist_zk.curve.scalar_mult(r, dist_zk.curve.G)
        input_list.append(point)

    # Create permutation
    permutation = [2, 0, 1]  # Shuffle
    output_list = [input_list[i] for i in permutation]

    print(f"Input list: {len(input_list)} elements")
    print(f"Output list: {len(output_list)} elements")
    print(f"Permutation: {permutation}")

    # Generate shuffle proof
    shuffle_proof = dist_zk.prove_shuffle(input_list, output_list, permutation)
    print(f"\nShuffle proof generated")

    # Verify shuffle proof
    valid = dist_zk.verify_shuffle(shuffle_proof, input_list, output_list)
    print(f"Shuffle proof valid: {'✓' if valid else '✗'}")

    # 5. zk-SNARK for Circuit
    print("\n\n5. zk-SNARK for Arithmetic Circuit:")
    print("-"*40)

    # Simple circuit: out = in1 * in2
    in1 = 7
    in2 = 6
    out = 42

    print(f"Circuit: out = in1 * in2")
    print(f"Inputs: {in1}, {in2}")
    print(f"Output: {out}")
    print(f"Correct: {out == in1 * in2}")

    # Setup
    proving_key, verification_key = dist_zk.zksnark.setup(None)
    print(f"\nzk-SNARK setup complete")
    print(f"  Proving key size: ~{len(str(proving_key))} characters")
    print(f"  Verification key size: ~{len(str(verification_key))} characters")

    # Generate proof
    zk_proof = dist_zk.zksnark.prove(proving_key, [in1, in2], out)
    print(f"\nzk-SNARK proof generated")
    print(f"  Proof size: ~{len(str(zk_proof))} characters")

    # Verify proof
    valid = dist_zk.zksnark.verify(verification_key, zk_proof, [in1, in2])
    print(f"zk-SNARK proof valid: {'✓' if valid else '✗'}")

def zk_proofs_mathematical_foundations():
    """
    Mathematical foundations of zero-knowledge proofs
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF ZERO-KNOWLEDGE PROOFS")
    print("="*70)

    print("\nDefinition 1: Interactive Proof System")
    print("  An interactive proof system for language L is a pair (P, V) where:")
    print("    1. Completeness: ∀x ∈ L, Pr[V accepts] ≥ 2/3")
    print("    2. Soundness: ∀x ∉ L, ∀P*, Pr[V accepts] ≤ 1/3")

    print("\nDefinition 2: Zero-Knowledge")
    print("  (P, V) is zero-knowledge if ∃ simulator S such that ∀x ∈ L:")
    print("    View_V[P(x) ↔ V(x)] ≈ S(x)")
    print("  where ≈ denotes computational indistinguishability")

    print("\nTheorem 1: Σ-protocols are Honest-Verifier Zero-Knowledge")
    print("Proof:")
    print("  1. Simulator chooses random challenge c and response s")
    print("  2. Computes commitment t = g^s * y^{-c}")
    print("  3. Output (t, c, s)")
    print("  4. Distribution identical to real protocol. ✓")

    print("\nTheorem 2: Fiat-Shamir Heuristic Security")
    print("Proof (ROM - Random Oracle Model):")
    print("  If hash function H is modeled as random oracle,")
    print("  then non-interactive protocol is secure")
    print("  assuming underlying Σ-protocol is secure. ✓")

    print("\nTheorem 3: zk-SNARK Knowledge Soundness")
    print("Proof (Knowledge of Exponent Assumption):")
    print("  If prover can produce valid proof,")
    print("  then they know witness w such that C(x, w) = 1")
    print("  under q-PKE (Power Knowledge of Exponent) assumption. ✓")

    print("\nTheorem 4: Bulletproofs Communication Complexity")
    print("Proof:")
    print("  Range proof size: O(log n) where n is bits")
    print("  Compared to O(n) for previous schemes")
    print("  Achieved through inner product argument. ✓")

def zk_applications_distributed_systems():
    """
    Applications of ZK proofs in distributed systems
    """
    print("\n" + "="*70)
    print("APPLICATIONS IN DISTRIBUTED SYSTEMS")
    print("="*70)

    applications = [
        ("Blockchain Privacy", """
        - Zcash: zk-SNARKs for private transactions
        - Monero: Ring signatures + Bulletproofs
        - Tornado Cash: Mixers using zk proofs
        """),

        ("Authentication & Authorization", """
        - Passwordless authentication
        - Attribute-based credentials
        - Anonymous credentials
        """),

        ("Verifiable Computation", """
        - Outsourced computation verification
        - Cloud computing integrity
        - Distributed machine learning
        """),

        ("Distributed Ledgers", """
        - Private smart contracts
        - Confidential assets
        - Scalable payment channels
        """),

        ("Voting Systems", """
        - End-to-end verifiable voting
        - Privacy-preserving tallying
        - Anonymous credentials for voters
        """),
    ]

    for title, description in applications:
        print(f"\n{title}:")
        print(description)

    print("\nPerformance Characteristics:")
    print("-"*40)

    characteristics = [
        ("Σ-protocols", "3 rounds", "O(1) proof size", "Fast", "Interactive"),
        ("Bulletproofs", "Non-interactive", "O(log n)", "Medium", "No trusted setup"),
        ("zk-SNARKs", "Non-interactive", "O(1)", "Slow setup", "Trusted setup"),
        ("zk-STARKs", "Non-interactive", "O(log² n)", "Fast", "No trusted setup"),
    ]

    print(f"{'Type':<15} {'Rounds':<15} {'Proof Size':<15} {'Performance':<15} {'Setup':<15}")
    print("-"*70)
    for typ, rounds, size, perf, setup in characteristics:
        print(f"{typ:<15} {rounds:<15} {size:<15} {perf:<15} {setup:<15}")

if __name__ == "__main__":
    demonstrate_zk_proofs()
    zk_proofs_mathematical_foundations()
    zk_applications_distributed_systems()

9. Distributed Tracing Theory

Mathematical Foundations of Distributed Tracing

Distributed tracing involves understanding causal relationships across services:

Mathematical Models:

Causal Ordering: Partial order based on happens-before relation (→)
Vector Clocks: Logical clocks that track causality
Span DAG: Directed Acyclic Graph representation of traces

Formal Definition:
A trace T = (S, ≤) where:

S = set of spans
≤ = happens-before relation (reflexive, antisymmetric, transitive)

Complete Python Implementation: OpenTelemetry-like Tracing

import uuid
import time
import threading
from typing import Dict, List, Set, Optional, Tuple, Any
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict, deque
import heapq
import random
import statistics
from datetime import datetime, timedelta

class TraceState:
    """
    W3C Trace Context state
    """

    def __init__(self, items: Dict[str, str] = None):
        self.items = items or {}

    def add(self, key: str, value: str):
        """Add key-value pair to trace state"""
        self.items[key] = value

    def get(self, key: str, default: str = None) -> Optional[str]:
        """Get value from trace state"""
        return self.items.get(key, default)

    def remove(self, key: str):
        """Remove key from trace state"""
        if key in self.items:
            del self.items[key]

    def to_header(self) -> str:
        """Convert to W3C trace state header format"""
        pairs = []
        for k, v in self.items.items():
            # Vendor key format: vendor@version=value
            pairs.append(f"{k}={v}")
        return ",".join(pairs)

    @classmethod
    def from_header(cls, header: str) -> 'TraceState':
        """Parse from W3C trace state header"""
        items = {}
        if header:
            for pair in header.split(','):
                if '=' in pair:
                    k, v = pair.split('=', 1)
                    items[k.strip()] = v.strip()
        return cls(items)

@dataclass
class SpanContext:
    """
    Context propagated across service boundaries
    """
    trace_id: str
    span_id: str
    trace_flags: int = 1  # Sampled flag
    trace_state: TraceState = field(default_factory=TraceState)
    is_remote: bool = False

    @property
    def is_valid(self) -> bool:
        """Check if context is valid"""
        return bool(self.trace_id and self.span_id)

    @property
    def is_sampled(self) -> bool:
        """Check if trace is sampled"""
        return bool(self.trace_flags & 1)

    def to_headers(self) -> Dict[str, str]:
        """Convert to W3C trace context headers"""
        headers = {
            'traceparent': f"00-{self.trace_id}-{self.span_id}-{self.trace_flags:02x}",
        }

        trace_state = self.trace_state.to_header()
        if trace_state:
            headers['tracestate'] = trace_state

        return headers

    @classmethod
    def from_headers(cls, headers: Dict[str, str]) -> 'SpanContext':
        """Create from W3C trace context headers"""
        traceparent = headers.get('traceparent', '')
        tracestate = headers.get('tracestate', '')

        if not traceparent:
            return cls.create_invalid()

        # Parse traceparent: version-trace-id-parent-id-flags
        parts = traceparent.split('-')
        if len(parts) != 4:
            return cls.create_invalid()

        version, trace_id, span_id, flags = parts

        try:
            trace_flags = int(flags, 16)
        except ValueError:
            return cls.create_invalid()

        trace_state = TraceState.from_header(tracestate)

        return cls(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags,
            trace_state=trace_state,
            is_remote=True
        )

    @classmethod
    def create_invalid(cls) -> 'SpanContext':
        """Create invalid context"""
        return cls(trace_id="", span_id="", trace_flags=0)

    @classmethod
    def generate(cls, sampled: bool = True) -> 'SpanContext':
        """Generate new context"""
        trace_id = uuid.uuid4().hex[:32]
        span_id = uuid.uuid4().hex[:16]
        trace_flags = 1 if sampled else 0

        return cls(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags
        )

class SpanKind(Enum):
    """Types of spans"""
    INTERNAL = "internal"
    SERVER = "server"
    CLIENT = "client"
    PRODUCER = "producer"
    CONSUMER = "consumer"

@dataclass
class Span:
    """
    Representation of a single operation
    """
    name: str
    context: SpanContext
    parent_context: Optional[SpanContext] = None
    kind: SpanKind = SpanKind.INTERNAL
    start_time: float = field(default_factory=time.time)
    end_time: Optional[float] = None
    attributes: Dict[str, Any] = field(default_factory=dict)
    events: List[Dict[str, Any]] = field(default_factory=list)
    status: Dict[str, Any] = field(default_factory=dict)
    links: List[SpanContext] = field(default_factory=list)

    def add_event(self, name: str, attributes: Dict[str, Any] = None):
        """Add event to span"""
        self.events.append({
            'name': name,
            'timestamp': time.time(),
            'attributes': attributes or {}
        })

    def set_attribute(self, key: str, value: Any):
        """Set span attribute"""
        self.attributes[key] = value

    def set_status(self, code: str, description: str = ""):
        """Set span status"""
        self.status = {'code': code, 'description': description}

    def end(self):
        """End the span"""
        if self.end_time is None:
            self.end_time = time.time()

    @property
    def duration(self) -> float:
        """Get span duration in seconds"""
        if self.end_time is None:
            return time.time() - self.start_time
        return self.end_time - self.start_time

    @property
    def is_ended(self) -> bool:
        """Check if span has ended"""
        return self.end_time is not None

    def to_dict(self) -> Dict[str, Any]:
        """Convert span to dictionary"""
        return {
            'name': self.name,
            'trace_id': self.context.trace_id,
            'span_id': self.context.span_id,
            'parent_span_id': self.parent_context.span_id if self.parent_context else None,
            'kind': self.kind.value,
            'start_time': self.start_time,
            'end_time': self.end_time,
            'duration': self.duration,
            'attributes': self.attributes,
            'events': self.events,
            'status': self.status,
            'links': [link.span_id for link in self.links]
        }

class Sampler:
    """
    Base class for trace sampling strategies
    """

    def should_sample(self, 
                     parent_context: Optional[SpanContext],
                     trace_id: str,
                     name: str,
                     kind: SpanKind,
                     attributes: Dict[str, Any]) -> Tuple[bool, List[Dict[str, Any]]]:
        """
        Decide whether to sample a span

        Returns: (sampled, attributes)
        """
        raise NotImplementedError

    def get_description(self) -> str:
        """Get sampler description"""
        raise NotImplementedError

class AlwaysOnSampler(Sampler):
    """Sample all traces"""

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        return True, []

    def get_description(self):
        return "AlwaysOnSampler"

class AlwaysOffSampler(Sampler):
    """Sample no traces"""

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        return False, []

    def get_description(self):
        return "AlwaysOffSampler"

class TraceIdRatioBasedSampler(Sampler):
    """Sample based on trace ID ratio"""

    def __init__(self, ratio: float):
        if not 0 <= ratio <= 1:
            raise ValueError("Ratio must be between 0 and 1")
        self.ratio = ratio

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        # Use trace ID to make deterministic sampling decision
        # Take first 8 bytes of trace ID as number
        trace_id_bytes = bytes.fromhex(trace_id[:16])
        trace_id_int = int.from_bytes(trace_id_bytes, 'big')

        # Use upper 56 bits for sampling (avoiding modulo bias)
        upper_bits = trace_id_int >> 8
        sampled = (upper_bits % 10000) < (self.ratio * 10000)

        return sampled, [{"sampler.ratio": self.ratio}]

    def get_description(self):
        return f"TraceIdRatioBased{{{self.ratio}}}"

class ParentBasedSampler(Sampler):
    """
    Sampler that respects parent's sampling decision
    """

    def __init__(self, 
                 root: Sampler,
                 remote_parent_sampled: Sampler,
                 remote_parent_not_sampled: Sampler,
                 local_parent_sampled: Sampler,
                 local_parent_not_sampled: Sampler):
        self.root = root
        self.remote_parent_sampled = remote_parent_sampled
        self.remote_parent_not_sampled = remote_parent_not_sampled
        self.local_parent_sampled = local_parent_sampled
        self.local_parent_not_sampled = local_parent_not_sampled

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        if not parent_context:
            # No parent - use root sampler
            return self.root.should_sample(parent_context, trace_id, name, kind, attributes)

        if parent_context.is_remote:
            if parent_context.is_sampled:
                return self.remote_parent_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
            else:
                return self.remote_parent_not_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
        else:
            if parent_context.is_sampled:
                return self.local_parent_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
            else:
                return self.local_parent_not_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)

    def get_description(self):
        return f"ParentBased{{root={self.root.get_description()}}}"

class AdaptiveSampler(Sampler):
    """
    Adaptive sampler that adjusts rate based on load
    """

    def __init__(self, 
                 target_rate: float = 10.0,  # Spans per second
                 adjustment_window: int = 60):  # Seconds
        self.target_rate = target_rate
        self.adjustment_window = adjustment_window
        self.current_ratio = 1.0
        self.span_counts = deque(maxlen=adjustment_window)
        self.last_adjustment = time.time()

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        # Update counts
        current_time = time.time()
        self.span_counts.append(current_time)

        # Adjust ratio periodically
        if current_time - self.last_adjustment > 1.0:  # Every second
            self._adjust_ratio()
            self.last_adjustment = current_time

        # Make sampling decision
        trace_id_bytes = bytes.fromhex(trace_id[:16])
        trace_id_int = int.from_bytes(trace_id_bytes, 'big')
        upper_bits = trace_id_int >> 8
        sampled = (upper_bits % 10000) < (self.current_ratio * 10000)

        return sampled, [{"sampler.adaptive.ratio": self.current_ratio}]

    def _adjust_ratio(self):
        """Adjust sampling ratio based on current load"""
        current_time = time.time()

        # Count spans in last window
        window_start = current_time - self.adjustment_window
        span_count = sum(1 for t in self.span_counts if t > window_start)

        actual_rate = span_count / self.adjustment_window

        # Adjust ratio to reach target rate
        if actual_rate > self.target_rate * 1.1:
            # Too many spans, reduce sampling
            self.current_ratio *= 0.9
        elif actual_rate < self.target_rate * 0.9:
            # Too few spans, increase sampling
            self.current_ratio = min(1.0, self.current_ratio * 1.1)

        # Clamp between 0.01 and 1.0
        self.current_ratio = max(0.01, min(1.0, self.current_ratio))

    def get_description(self):
        return f"AdaptiveSampler{{target={self.target_rate}, ratio={self.current_ratio:.3f}}}"

class Tracer:
    """
    Main tracer implementation
    """

    def __init__(self, 
                 name: str,
                 sampler: Sampler = None,
                 max_spans_per_trace: int = 1000):
        self.name = name
        self.sampler = sampler or AlwaysOnSampler()
        self.max_spans_per_trace = max_spans_per_trace

        # Active spans
        self.active_spans = defaultdict(list)  # trace_id -> list of spans

        # Statistics
        self.stats = {
            'spans_created': 0,
            'spans_ended': 0,
            'traces_sampled': 0,
            'traces_dropped': 0
        }

        # Exporters
        self.exporters = []

    def start_span(self, 
                  name: str,
                  context: Optional[SpanContext] = None,
                  kind: SpanKind = SpanKind.INTERNAL,
                  attributes: Dict[str, Any] = None,
                  links: List[SpanContext] = None,
                  start_time: Optional[float] = None) -> Optional[Span]:
        """
        Start a new span
        """
        attributes = attributes or {}
        links = links or []

        # Determine parent context
        parent_context = context

        # Generate trace ID if not provided
        if parent_context and parent_context.is_valid:
            trace_id = parent_context.trace_id
        else:
            trace_id = uuid.uuid4().hex[:32]

        # Check if we should sample
        sampled, sampler_attrs = self.sampler.should_sample(
            parent_context, trace_id, name, kind, attributes)

        if not sampled:
            self.stats['traces_dropped'] += 1
            return None

        self.stats['traces_sampled'] += 1

        # Generate span context
        span_id = uuid.uuid4().hex[:16]
        trace_flags = 1 if sampled else 0

        span_context = SpanContext(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags
        )

        # Create span
        span = Span(
            name=name,
            context=span_context,
            parent_context=parent_context,
            kind=kind,
            start_time=start_time or time.time(),
            attributes={**attributes, **{k: v for k, v in sampler_attrs}},
            links=links
        )

        # Add to active spans
        self.active_spans[trace_id].append(span)

        # Check trace size limit
        if len(self.active_spans[trace_id]) > self.max_spans_per_trace:
            self._cleanup_oldest_span(trace_id)

        self.stats['spans_created'] += 1

        return span

    def _cleanup_oldest_span(self, trace_id: str):
        """Remove oldest span from trace"""
        if trace_id in self.active_spans and self.active_spans[trace_id]:
            # Find and remove oldest span
            oldest_idx = 0
            oldest_time = self.active_spans[trace_id][0].start_time

            for i, span in enumerate(self.active_spans[trace_id][1:], 1):
                if span.start_time < oldest_time:
                    oldest_time = span.start_time
                    oldest_idx = i

            removed = self.active_spans[trace_id].pop(oldest_idx)
            if not self.active_spans[trace_id]:
                del self.active_spans[trace_id]

    def end_span(self, span: Span):
        """End a span"""
        if span.is_ended:
            return

        span.end()
        self.stats['spans_ended'] += 1

        # Check if trace is complete
        trace_id = span.context.trace_id
        if trace_id in self.active_spans:
            # Remove this span from active
            self.active_spans[trace_id] = [s for s in self.active_spans[trace_id] 
                                          if s.context.span_id != span.context.span_id]

            # If no more active spans, trace is complete
            if not self.active_spans[trace_id]:
                del self.active_spans[trace_id]
                self._export_trace([span])  # Simplified - would export all spans

    def get_active_traces(self) -> Dict[str, List[Span]]:
        """Get all active traces"""
        return dict(self.active_spans)

    def get_statistics(self) -> Dict[str, Any]:
        """Get tracer statistics"""
        return {
            **self.stats,
            'active_traces': len(self.active_spans),
            'active_spans': sum(len(spans) for spans in self.active_spans.values()),
            'sampler': self.sampler.get_description()
        }

    def add_exporter(self, exporter):
        """Add span exporter"""
        self.exporters.append(exporter)

    def _export_trace(self, spans: List[Span]):
        """Export completed trace"""
        for exporter in self.exporters:
            try:
                exporter.export(spans)
            except Exception as e:
                print(f"Exporter error: {e}")

class TraceAnalyzer:
    """
    Analyze traces for performance and issues
    """

    def __init__(self):
        self.traces = []  # List of completed traces
        self.metrics = defaultdict(list)

    def add_trace(self, spans: List[Span]):
        """Add completed trace for analysis"""
        self.traces.append(spans)

        # Update metrics
        self._update_metrics(spans)

    def _update_metrics(self, spans: List[Span]):
        """Update metrics from trace"""
        if not spans:
            return

        # Trace duration
        start_times = [s.start_time for s in spans]
        end_times = [s.end_time for s in spans if s.end_time]

        if start_times and end_times:
            trace_start = min(start_times)
            trace_end = max(end_times)
            trace_duration = trace_end - trace_start

            self.metrics['trace_duration'].append(trace_duration)

        # Span durations
        for span in spans:
            if span.end_time:
                self.metrics['span_duration'].append(span.duration)

        # Spans per trace
        self.metrics['spans_per_trace'].append(len(spans))

    def get_percentiles(self, metric: str, percentiles: List[float] = None) -> Dict[float, float]:
        """Calculate percentiles for metric"""
        if metric not in self.metrics or not self.metrics[metric]:
            return {}

        values = self.metrics[metric]
        percentiles = percentiles or [0.5, 0.75, 0.9, 0.95, 0.99]

        results = {}
        for p in percentiles:
            idx = int(p * len(values))
            sorted_values = sorted(values)
            results[p] = sorted_values[idx] if idx < len(sorted_values) else sorted_values[-1]

        return results

    def analyze_critical_path(self, spans: List[Span]) -> List[Span]:
        """
        Find critical path (longest path) in trace
        """
        if not spans:
            return []

        # Build graph
        span_dict = {s.context.span_id: s for s in spans}
        children = defaultdict(list)

        for span in spans:
            if span.parent_context and span.parent_context.span_id in span_dict:
                parent_id = span.parent_context.span_id
                children[parent_id].append(span.context.span_id)

        # Find root spans
        root_spans = [s for s in spans if not s.parent_context or 
                     s.parent_context.span_id not in span_dict]

        if not root_spans:
            return []

        # DFS to find longest path
        def dfs(span_id: str, path: List[str], path_duration: float) -> Tuple[List[str], float]:
            span = span_dict[span_id]
            current_duration = path_duration + span.duration

            if not children[span_id]:
                return path + [span_id], current_duration

            best_path = []
            best_duration = 0

            for child_id in children[span_id]:
                child_path, child_duration = dfs(child_id, path + [span_id], current_duration)
                if child_duration > best_duration:
                    best_path = child_path
                    best_duration = child_duration

            return best_path, best_duration

        # Find critical path from each root
        critical_path = []
        critical_duration = 0

        for root in root_spans:
            path, duration = dfs(root.context.span_id, [], 0)
            if duration > critical_duration:
                critical_path = path
                critical_duration = duration

        return [span_dict[span_id] for span_id in critical_path]

    def detect_anomalies(self, spans: List[Span], 
                        threshold_stddev: float = 2.0) -> List[Dict[str, Any]]:
        """
        Detect anomalous spans in trace
        """
        anomalies = []

        if not spans:
            return anomalies

        # Calculate statistics for similar spans
        span_groups = defaultdict(list)
        for span in spans:
            span_groups[span.name].append(span.duration)

        for name, durations in span_groups.items():
            if len(durations) < 3:
                continue

            mean = statistics.mean(durations)
            stdev = statistics.stdev(durations) if len(durations) > 1 else 0

            # Find spans more than threshold_stddev from mean
            for span in spans:
                if span.name == name and span.end_time:
                    z_score = abs(span.duration - mean) / stdev if stdev > 0 else 0
                    if z_score > threshold_stddev:
                        anomalies.append({
                            'span': span,
                            'name': name,
                            'duration': span.duration,
                            'mean_duration': mean,
                            'z_score': z_score,
                            'anomaly_type': 'duration'
                        })

        return anomalies

    def visualize_trace(self, spans: List[Span], max_width: int = 80):
        """
        Create ASCII visualization of trace
        """
        if not spans:
            return ""

        # Sort spans by start time
        sorted_spans = sorted(spans, key=lambda s: s.start_time)

        # Find time range
        start_times = [s.start_time for s in sorted_spans]
        end_times = [s.end_time for s in sorted_spans if s.end_time]

        if not end_times:
            return "Trace not complete"

        min_time = min(start_times)
        max_time = max(end_times)
        time_range = max_time - min_time

        if time_range == 0:
            time_range = 1

        # Build visualization
        lines = []
        lines.append(f"Trace Duration: {time_range:.3f}s")
        lines.append("=" * max_width)

        for span in sorted_spans:
            # Calculate position and width
            start_pos = int((span.start_time - min_time) / time_range * max_width)
            duration = span.duration if span.end_time else (time.time() - span.start_time)
            width = max(1, int(duration / time_range * max_width))

            # Truncate name if needed
            name = span.name
            if len(name) > width - 2:
                name = name[:width-5] + "..."

            # Create bar
            bar = " " * start_pos + "[" + name.center(width-2) + "]"
            lines.append(bar)

            # Add duration info
            lines.append(f"  {span.name}: {duration:.3f}s")

        return "\n".join(lines)

def demonstrate_distributed_tracing():
    """
    Complete demonstration of distributed tracing
    """
    print("="*70)
    print("DISTRIBUTED TRACING THEORY AND IMPLEMENTATION")
    print("="*70)

    # 1. Create Tracer with Adaptive Sampling
    print("\n1. Creating Tracer with Adaptive Sampling:")
    print("-"*40)

    adaptive_sampler = AdaptiveSampler(target_rate=5.0)
    tracer = Tracer(name="example-service", sampler=adaptive_sampler)

    print(f"Tracer created: {tracer.name}")
    print(f"Sampler: {tracer.sampler.get_description()}")

    # 2. Simulate Distributed Trace
    print("\n\n2. Simulating Distributed Trace Across Services:")
    print("-"*40)

    # Service A: Receives request
    print("\nService A (HTTP Server):")
    print("  Receives request from client")

    # Create root span
    root_span = tracer.start_span(
        name="HTTP GET /api/users",
        kind=SpanKind.SERVER,
        attributes={
            "http.method": "GET",
            "http.route": "/api/users",
            "http.url": "http://example.com/api/users",
            "net.peer.ip": "192.168.1.100"
        }
    )

    if root_span:
        print(f"  Created root span: {root_span.name}")
        print(f"  Trace ID: {root_span.context.trace_id[:16]}...")
        print(f"  Span ID: {root_span.context.span_id}")

        # Service A calls Service B (Database)
        print("\n  Service A calls Service B (Database):")

        # Extract context for propagation
        headers = root_span.context.to_headers()
        print(f"  Propagating context headers: {headers}")

        # Service B receives context
        print("\nService B (Database):")
        print("  Receives call from Service A")

        # Create child span
        child_span = tracer.start_span(
            name="Database Query",
            context=SpanContext.from_headers(headers),
            kind=SpanKind.CLIENT,
            attributes={
                "db.system": "postgresql",
                "db.operation": "SELECT",
                "db.query": "SELECT * FROM users"
            }
        )

        if child_span:
            print(f"  Created child span: {child_span.name}")
            print(f"  Parent Span ID: {child_span.parent_context.span_id}")

            # Simulate database work
            time.sleep(0.05)

            # End child span
            child_span.end()
            tracer.end_span(child_span)
            print(f"  Database query completed: {child_span.duration:.3f}s")

        # Service A continues processing
        print("\nService A (continues):")

        # Add event to root span
        root_span.add_event("database_query_completed")

        # Simulate more work
        time.sleep(0.02)

        # End root span
        root_span.end()
        tracer.end_span(root_span)
        print(f"  Request completed: {root_span.duration:.3f}s")

    # 3. Sampling Strategies Comparison
    print("\n\n3. Sampling Strategies Comparison:")
    print("-"*40)

    samplers = [
        ("Always On", AlwaysOnSampler()),
        ("Always Off", AlwaysOffSampler()),
        ("10% Ratio", TraceIdRatioBasedSampler(0.1)),
        ("Adaptive (5/s)", AdaptiveSampler(target_rate=5.0)),
    ]

    print(f"\n{'Sampler':<20} {'Description':<30} {'Sampled %':<10}")
    print("-"*70)

    # Test each sampler
    test_trace_id = uuid.uuid4().hex[:32]

    for name, sampler in samplers:
        sampled_count = 0
        total_tests = 1000

        for _ in range(total_tests):
            sampled, _ = sampler.should_sample(
                None, test_trace_id, "test-span", SpanKind.INTERNAL, {})
            if sampled:
                sampled_count += 1

        percentage = sampled_count / total_tests * 100
        print(f"{name:<20} {sampler.get_description():<30} {percentage:>8.1f}%")

    # 4. Trace Analysis
    print("\n\n4. Trace Analysis and Critical Path:")
    print("-"*40)

    analyzer = TraceAnalyzer()

    # Create complex trace for analysis
    spans = []

    # Root span
    root = Span(
        name="ProcessOrder",
        context=SpanContext.generate(),
        start_time=time.time()
    )
    spans.append(root)

    # Child spans
    inventory_check = Span(
        name="CheckInventory",
        context=SpanContext.generate(),
        parent_context=root.context,
        start_time=root.start_time + 0.01
    )
    inventory_check.end_time = inventory_check.start_time + 0.05
    spans.append(inventory_check)

    payment_processing = Span(
        name="ProcessPayment",
        context=SpanContext.generate(),
        parent_context=root.context,
        start_time=root.start_time + 0.02
    )
    payment_processing.end_time = payment_processing.start_time + 0.1  # Slow!
    spans.append(payment_processing)

    shipping_calculation = Span(
        name="CalculateShipping",
        context=SpanContext.generate(),
        parent_context=inventory_check.context,
        start_time=inventory_check.end_time + 0.01
    )
    shipping_calculation.end_time = shipping_calculation.start_time + 0.03
    spans.append(shipping_calculation)

    root.end_time = max(s.end_time for s in spans if s.end_time)

    # Analyze
    analyzer.add_trace(spans)

    # Find critical path
    critical_path = analyzer.analyze_critical_path(spans)
    print("\nCritical Path (longest duration):")
    for span in critical_path:
        print(f"  {span.name}: {span.duration:.3f}s")

    # Detect anomalies
    anomalies = analyzer.detect_anomalies(spans)
    if anomalies:
        print("\nDetected Anomalies:")
        for anomaly in anomalies:
            print(f"  {anomaly['name']}: {anomaly['duration']:.3f}s "
                  f"(expected ~{anomaly['mean_duration']:.3f}s, "
                  f"z-score={anomaly['z_score']:.1f})")

    # Visualize trace
    print("\nTrace Visualization:")
    visualization = analyzer.visualize_trace(spans, max_width=60)
    print(visualization)

    # 5. W3C Trace Context Propagation
    print("\n\n5. W3C Trace Context Propagation:")
    print("-"*40)

    # Create context
    context = SpanContext.generate(sampled=True)
    context.trace_state.add("vendor1", "value1")
    context.trace_state.add("vendor2", "value2")

    print("Original Context:")
    print(f"  Trace ID: {context.trace_id}")
    print(f"  Span ID: {context.span_id}")
    print(f"  Sampled: {context.is_sampled}")
    print(f"  Trace State: {context.trace_state.to_header()}")

    # Convert to headers
    headers = context.to_headers()
    print("\nHTTP Headers for Propagation:")
    for k, v in headers.items():
        print(f"  {k}: {v}")

    # Parse from headers
    parsed = SpanContext.from_headers(headers)
    print("\nParsed Context:")
    print(f"  Trace ID: {parsed.trace_id}")
    print(f"  Span ID: {parsed.span_id}")
    print(f"  Sampled: {parsed.is_sampled}")
    print(f"  Trace State: {parsed.trace_state.to_header()}")
    print(f"  Valid: {parsed.is_valid}")

    # 6. Statistics and Monitoring
    print("\n\n6. Tracing Statistics:")
    print("-"*40)

    stats = tracer.get_statistics()
    for key, value in stats.items():
        print(f"  {key}: {value}")

def tracing_mathematical_foundations():
    """
    Mathematical foundations of distributed tracing
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF DISTRIBUTED TRACING")
    print("="*70)

    print("\nDefinition 1: Happens-Before Relation (→)")
    print("  For events a, b in distributed system:")
    print("    1. If a and b are in same process and a occurs before b, then a → b")
    print("    2. If a is sending message m and b is receiving m, then a → b")
    print("    3. If a → b and b → c, then a → c (transitivity)")

    print("\nTheorem 1: Vector Clocks Capture Causality")
    print("Proof:")
    print("  For vector clocks V, W:")
    print("    V < W iff ∀i: V[i] ≤ W[i] and ∃j: V[j] < W[j]")
    print("  Then: a → b iff VC(a) < VC(b)")
    print("  Proof by induction on causal paths. ✓")

    print("\nTheorem 2: Sampling Rate and Error Bound")
    print("  For sampling rate p, estimation error ε:")
    print("    Pr(|X̂ - X| ≥ εX) ≤ 2 exp(-pε²X/3)")
    print("  where X is true value, X̂ is sampled estimate")
    print("  Proof using Chernoff bound. ✓")

    print("\nTheorem 3: Critical Path Identifies Bottleneck")
    print("Proof:")
    print("  Let G = (V, E) be trace DAG with weights = durations")
    print("  Critical path = longest path from source to sink")
    print("  By definition, reducing any node on critical path")
    print("  reduces total trace duration. ✓")

    print("\nTheorem 4: Optimal Sampling for Rare Events")
    print("  For rare event with probability q, optimal sampling:")
    print("    p* = min(1, √(N/qT))")
    print("  where N is sampling cost, T is trace budget")
    print("  Proof via Lagrange multiplier. ✓")

def trace_analysis_algorithms():
    """
    Algorithms for trace analysis
    """
    print("\n" + "="*70)
    print("TRACE ANALYSIS ALGORITHMS")
    print("="*70)

    algorithms = [
        ("Critical Path Analysis", """
        Input: Trace DAG G = (V, E) with weights w(v)
        Algorithm:
          1. Topological sort of vertices
          2. For each vertex v in topological order:
              dist[v] = max(dist[u] + w(u) for u in predecessors(v))
              prev[v] = argmax_u(dist[u] + w(u))
          3. Reconstruct path from sink to source using prev[]
        Complexity: O(|V| + |E|)
        """),

        ("Anomaly Detection (Statistical)", """
        Input: Span durations for operation O
        Algorithm:
          1. Collect historical durations: X = {x₁, ..., xₙ}
          2. Compute μ = mean(X), σ = stddev(X)
          3. For new observation x:
              z = |x - μ| / σ
              If z > threshold (e.g., 3), flag as anomaly
          4. Use robust statistics (median, MAD) for outliers
        """),

        ("Trace Compression (Lossy)", """
        Input: Trace T with n spans
        Algorithm:
          1. Build span similarity graph
          2. Cluster similar spans (same name, similar attributes)
          3. For each cluster, keep k representative spans
          4. Reconstruct approximate trace
        Compression ratio: O(n/k)
        """),

        ("Root Cause Analysis", """
        Input: Failed trace T_f, successful traces T_s
        Algorithm:
          1. Extract features from traces (durations, paths, attributes)
          2. Train classifier or compute similarity
          3. Find most divergent spans in T_f
          4. Rank potential root causes by divergence score
        Accuracy depends on training data
        """),
    ]

    for name, description in algorithms:
        print(f"\n{name}:")
        print(description)

if __name__ == "__main__":
    demonstrate_distributed_tracing()
    tracing_mathematical_foundations()
    trace_analysis_algorithms()

This is the beginning of a completely comprehensive guide to distributed systems theory and practice.
Share your ideas, comments and your questions with us by your comments here! We and the community will answer you 24/7!
I hope you enjoy, have nice times!

Distributed Systems & Networking: Time, Order, and Coordination. Beyond Simple Timestamps

Javad — Tue, 06 Jan 2026 22:43:53 +0000

Hey Dev Community!
Welcome to the deepest journey of Distributed Systems ever exists!

🚀 Introduction: Why Time is the Hardest Problem

Welcome back, distributed systems architects! ⚡

If you've ever tried to debug a race condition, wondered why your database timestamps don't match, or struggled with replica synchronization, you've encountered the fundamental challenge of time in distributed systems.

Today, we're diving deep into the theoretical foundations of time, ordering, and coordination. This isn't just about clocks—it's about causality, consistency, and the very nature of distributed computation.

🔢 1. Logical Clocks: Lamport & Vector Clocks

Lamport Clocks: The Foundation of Logical Time

Leslie Lamport's seminal 1978 paper introduced the idea that we don't need real-time clocks—we just need to agree on happens-before relationships.

class LamportClock:
    def __init__(self, node_id):
        self.node_id = node_id
        self.counter = 0

    def increment(self):
        """Increment counter for local event"""
        self.counter += 1
        return self.counter

    def send_event(self):
        """Increment and return timestamp for sending message"""
        self.counter += 1
        return self.counter

    def receive_event(self, received_timestamp):
        """Update clock based on received message timestamp"""
        self.counter = max(self.counter, received_timestamp) + 1
        return self.counter

    def compare(self, timestamp_a, timestamp_b):
        """
        Compare two Lamport timestamps
        Returns: -1 if a < b, 0 if concurrent, 1 if a > b
        """
        if timestamp_a < timestamp_b:
            return -1
        elif timestamp_a > timestamp_b:
            return 1
        else:
            # Same timestamp, use node_id as tiebreaker
            return 0 if self.node_id == self.node_id else 1

    def happens_before(self, event_a, event_b):
        """
        Check if event_a happens before event_b
        Based on: a → b if:
        1. a and b are on same process and a comes before b
        2. a is send and b is receive of same message
        3. Transitive: ∃ c such that a → c and c → b
        """
        # Implementation of the happens-before relation
        pass

# Example: Distributed chat application
class DistributedChat:
    def __init__(self, user_id):
        self.clock = LamportClock(user_id)
        self.messages = []

    def send_message(self, content):
        timestamp = self.clock.send_event()
        message = {
            'content': content,
            'sender': self.clock.node_id,
            'timestamp': timestamp,
            'lamport_time': timestamp
        }
        self.messages.append(message)
        return message

    def receive_message(self, message):
        # Update clock with received timestamp
        self.clock.receive_event(message['lamport_time'])

        # Insert in happens-before order
        self._insert_ordered(message)

    def _insert_ordered(self, message):
        # Insert message maintaining causal order
        for i, existing in enumerate(self.messages):
            if self.clock.compare(message['lamport_time'], 
                                 existing['lamport_time']) < 0:
                self.messages.insert(i, message)
                return
        self.messages.append(message)

Key Insight: Lamport clocks guarantee that if event A happens before event B, then A's timestamp < B's timestamp. But the converse isn't true—equal timestamps don't guarantee concurrency detection.

Vector Clocks: Capturing Causality Precisely

Vector clocks extend Lamport clocks to detect concurrent events.

class VectorClock:
    def __init__(self, node_id, num_nodes):
        self.node_id = node_id
        self.num_nodes = num_nodes
        self.vector = [0] * num_nodes

    def increment(self):
        """Increment own component"""
        self.vector[self.node_id] += 1
        return self.vector.copy()

    def update(self, received_vector):
        """Merge with received vector clock (element-wise max)"""
        for i in range(self.num_nodes):
            self.vector[i] = max(self.vector[i], received_vector[i])
        self.vector[self.node_id] += 1  # Increment for receive event

    def compare(self, other_vector):
        """
        Compare two vector clocks
        Returns: 
          -1 if self < other (happens before)
           0 if concurrent
           1 if self > other (happens after)
           2 if equal
        """
        less = False
        greater = False

        for i in range(self.num_nodes):
            if self.vector[i] < other_vector[i]:
                less = True
            elif self.vector[i] > other_vector[i]:
                greater = True

        if less and not greater:
            return -1  # self happens before other
        elif greater and not less:
            return 1   # self happens after other
        elif not less and not greater:
            return 2   # equal
        else:
            return 0   # concurrent

    def is_causally_related(self, event_a, event_b):
        """Check if event_a causally precedes event_b"""
        return self.compare(event_a['vector'], event_b['vector']) == -1

    def get_concurrent_events(self, events):
        """Find all events concurrent with given event"""
        concurrent = []
        for event in events:
            if self.compare(event['vector'], self.vector) == 0:
                concurrent.append(event)
        return concurrent

# Example: Version conflicts in distributed database
class VersionedDatabase:
    def __init__(self, node_id, num_nodes):
        self.clock = VectorClock(node_id, num_nodes)
        self.data = {}

    def write(self, key, value):
        # Increment vector clock for write
        timestamp = self.clock.increment()

        version = {
            'value': value,
            'vector': timestamp.copy(),
            'node': self.clock.node_id
        }

        if key not in self.data:
            self.data[key] = []

        # Check for conflicts with existing versions
        conflicts = []
        for existing in self.data[key]:
            comp = self.clock.compare(timestamp, existing['vector'])
            if comp == 0:  # Concurrent
                conflicts.append(existing)

        if conflicts:
            # Handle concurrent writes (application-specific)
            version = self._resolve_conflict(version, conflicts)

        self.data[key].append(version)
        return version

    def read(self, key):
        # Return all non-obsolete versions
        if key not in self.data:
            return []

        # Filter out versions that are definitely older
        current_versions = []
        for version in self.data[key]:
            # Check if this version is not dominated by any other
            is_current = True
            for other in self.data[key]:
                if (version != other and 
                    self.clock.compare(version['vector'], other['vector']) == -1):
                    is_current = False
                    break

            if is_current:
                current_versions.append(version)

        return current_versions

    def _resolve_conflict(self, new_version, conflicts):
        # Example conflict resolution: last writer wins with tie-breaker
        # In practice, use application-specific logic
        all_versions = conflicts + [new_version]
        # Sort by vector clock (partial order)
        all_versions.sort(key=lambda v: tuple(v['vector']), reverse=True)
        return all_versions[0]

Vector Clock Properties:

Causality Detection: Precisely identifies concurrent events
Space Complexity: O(N) where N is number of nodes
Merge Operation: Element-wise maximum (commutative, associative)
Garbage Collection: Challenge in long-running systems

🕰️ 2. Hybrid Logical Clocks: Best of Both Worlds

Hybrid Logical Clocks (HLC) combine physical timestamps with logical counters to provide monotonicity and bounded skew.

import time
import struct
from typing import Tuple

class HybridLogicalClock:
    """
    HLC = (physical_time, logical_counter, node_id)

    Properties:
    1. Monotonic: Always increases
    2. Bounded skew: |HLC - physical_time| <= MAX_OFFSET
    3. Causal ordering: If event A → B, then HLC(A) < HLC(B)
    """

    def __init__(self, node_id: int, max_offset: int = 10):
        self.node_id = node_id
        self.max_offset = max_offset  # Maximum allowed clock skew (ms)

        # Current HLC state
        self.pt = 0  # Physical time component (milliseconds)
        self.l = 0   # Logical counter

    def now(self) -> Tuple[int, int, int]:
        """Get current HLC timestamp"""
        current_pt = self._get_physical_time()

        if current_pt > self.pt:
            # Physical time has advanced
            self.pt = current_pt
            self.l = 0
        else:
            # Same physical time, increment logical counter
            self.l += 1

        return (self.pt, self.l, self.node_id)

    def send(self) -> Tuple[int, int, int]:
        """HLC for sending a message"""
        return self.now()

    def receive(self, received_hlc: Tuple[int, int, int]) -> Tuple[int, int, int]:
        """Update HLC based on received timestamp"""
        received_pt, received_l, received_node = received_hlc
        current_pt = self._get_physical_time()

        # Update rule for HLC
        if current_pt > max(self.pt, received_pt):
            # Our physical time is ahead of both
            self.pt = current_pt
            self.l = 0
        elif received_pt > self.pt:
            # Received timestamp is ahead
            self.pt = received_pt
            self.l = received_l + 1
        elif received_pt == self.pt:
            # Same physical time
            self.l = max(self.l, received_l) + 1
        else:
            # Our clock is ahead of received
            self.l += 1

        # Bound check
        if abs(self.pt - current_pt) > self.max_offset:
            raise ClockSkewError(f"Clock skew {abs(self.pt - current_pt)} > {self.max_offset}")

        return (self.pt, self.l, self.node_id)

    def _get_physical_time(self) -> int:
        """Get current physical time in milliseconds"""
        return int(time.time() * 1000)

    def compare(self, hlc1: Tuple[int, int, int], hlc2: Tuple[int, int, int]) -> int:
        """Compare two HLC timestamps"""
        pt1, l1, n1 = hlc1
        pt2, l2, n2 = hlc2

        if pt1 < pt2:
            return -1
        elif pt1 > pt2:
            return 1
        else:
            # Same physical time
            if l1 < l2:
                return -1
            elif l1 > l2:
                return 1
            else:
                # Same logical counter
                if n1 < n2:
                    return -1
                elif n1 > n2:
                    return 1
                else:
                    return 0  # Equal

    def to_bytes(self, hlc: Tuple[int, int, int]) -> bytes:
        """Serialize HLC to bytes for storage/transmission"""
        pt, l, node = hlc
        return struct.pack('!QII', pt, l, node)

    def from_bytes(self, data: bytes) -> Tuple[int, int, int]:
        """Deserialize HLC from bytes"""
        return struct.unpack('!QII', data)

class ClockSkewError(Exception):
    pass

# Example: Distributed transaction logging
class DistributedTransactionLog:
    def __init__(self, node_id):
        self.hlc = HybridLogicalClock(node_id)
        self.log = []

    def log_transaction(self, transaction):
        timestamp = self.hlc.now()
        entry = {
            'transaction': transaction,
            'hlc': timestamp,
            'physical_time': time.time()
        }
        self.log.append(entry)

        # Sort log by HLC (causal order)
        self.log.sort(key=lambda x: x['hlc'])

        return timestamp

    def replicate_log(self, remote_entries):
        """Merge logs from another node"""
        for entry in remote_entries:
            # Update HLC with received timestamp
            self.hlc.receive(entry['hlc'])

            # Add to log if not already present
            if not any(self.hlc.compare(e['hlc'], entry['hlc']) == 0 
                      for e in self.log):
                self.log.append(entry)

        # Resort log
        self.log.sort(key=lambda x: x['hlc'])

    def get_ordered_transactions(self):
        """Get transactions in causal order"""
        return [entry['transaction'] for entry in self.log]

HLC Advantages:

Monotonic: Always increases
Causal Ordering: Preserves happens-before
Bounded Skew: Stays close to physical time
Compact: Fixed size (12 bytes typical)
No Synchronization: No need for clock synchronization protocols

⚛️ 3. TrueTime: Google Spanner's Atomic Precision

TrueTime is Google's solution for globally consistent timestamps using atomic clocks and GPS.

import time
import random
from dataclasses import dataclass
from typing import Tuple

@dataclass
class TrueTimeInterval:
    """
    TrueTime represents time as an interval [earliest, latest]
    with guaranteed error bounds
    """
    earliest: float  # Lower bound (seconds since epoch)
    latest: float    # Upper bound

    def midpoint(self) -> float:
        return (self.earliest + self.latest) / 2

    def error(self) -> float:
        return self.latest - self.earliest

    def contains(self, timestamp: float) -> bool:
        return self.earliest <= timestamp <= self.latest

    def definitely_before(self, other: 'TrueTimeInterval') -> bool:
        return self.latest < other.earliest

    def definitely_after(self, other: 'TrueTimeInterval') -> bool:
        return self.earliest > other.latest

    def concurrent(self, other: 'TrueTimeInterval') -> bool:
        return not (self.definitely_before(other) or 
                   self.definitely_after(other))

class TrueTimeAPI:
    """
    Simulated TrueTime API as used in Google Spanner

    Real implementation uses:
    - Atomic clocks (cesium, rubidium)
    - GPS receivers
    - Time master servers
    """

    def __init__(self, epsilon: float = 0.001):
        """
        epsilon: maximum clock uncertainty (1ms default)
        In production: 1-7ms for Spanner
        """
        self.epsilon = epsilon
        self.reference_time = time.time()

        # Simulate clock drift
        self.drift_rate = random.uniform(-0.0001, 0.0001)  # ±100ppm

    def now(self) -> TrueTimeInterval:
        """
        Returns: [earliest, latest] with |latest - earliest| <= 2ε
        """
        current = self._get_current_time()
        error = self._get_current_error()

        earliest = current - error
        latest = current + error

        return TrueTimeInterval(earliest, latest)

    def _get_current_time(self) -> float:
        """Get current time with simulated drift"""
        elapsed = time.time() - self.reference_time
        drift = elapsed * self.drift_rate
        return time.time() + drift

    def _get_current_error(self) -> float:
        """
        Calculate current error bound
        Components:
        1. Clock uncertainty (ε)
        2. Message delay uncertainty
        3. Master clock synchronization error
        """
        base_error = self.epsilon

        # Simulate additional error sources
        message_delay_error = random.uniform(0, 0.0005)  # Up to 0.5ms
        sync_error = random.uniform(0, 0.0002)          # Up to 0.2ms

        return base_error + message_delay_error + sync_error

    def after(self, timestamp: float) -> TrueTimeInterval:
        """
        Returns a timestamp that is guaranteed to be after the given time
        Used for commit waits in Spanner
        """
        now_interval = self.now()

        if timestamp < now_interval.latest:
            # Already past the timestamp
            return self.now()
        else:
            # Need to wait
            wait_time = timestamp - now_interval.midpoint()
            if wait_time > 0:
                time.sleep(wait_time)

            return self.now()

    def sleep(self, duration: float) -> TrueTimeInterval:
        """Sleep for at least duration, return timestamp after sleep"""
        time.sleep(duration + 2 * self.epsilon)  # Wait extra for safety
        return self.now()

# Example: Spanner-style distributed transactions
class SpannerTransaction:
    """
    Two-phase commit with TrueTime for global ordering

    Key idea: Use TrueTime to assign commit timestamps
    that respect causal order across datacenters
    """

    def __init__(self, truetime: TrueTimeAPI):
        self.truetime = truetime
        self.participants = []
        self.start_time = self.truetime.now()

    def prepare(self) -> bool:
        """Phase 1: Prepare all participants"""
        prepared = []
        for participant in self.participants:
            try:
                prepared.append(participant.prepare())
            except Exception as e:
                # Abort on any failure
                self._abort()
                return False

        return all(prepared)

    def commit(self) -> float:
        """Phase 2: Commit with TrueTime timestamp"""
        if not self.prepare():
            raise TransactionError("Prepare failed")

        # Get commit timestamp with uncertainty
        commit_interval = self.truetime.now()

        # Ensure this commit is definitely after start
        while not commit_interval.definitely_after(self.start_time):
            commit_interval = self.truetime.after(self.start_time.latest)

        # Commit timestamp is the midpoint
        commit_timestamp = commit_interval.midpoint()

        # Wait until commit timestamp is definitely in the past
        self.truetime.after(commit_timestamp)

        # Now safe to commit at all participants
        for participant in self.participants:
            participant.commit(commit_timestamp)

        return commit_timestamp

    def _abort(self):
        """Rollback all participants"""
        for participant in self.participants:
            participant.rollback()

class TransactionError(Exception):
    pass

# Example: Global schema change with TrueTime
class GlobalSchemaChange:
    """
    Using TrueTime to coordinate schema changes across datacenters
    """

    def __init__(self, truetime: TrueTimeAPI):
        self.truetime = truetime
        self.change_time = None

    def schedule_change(self, change_id: str, future_time: float):
        """
        Schedule schema change for a future time
        All datacenters will apply change at same logical time
        """
        # Get current time interval
        now = self.truetime.now()

        # Ensure future_time is definitely in the future
        if future_time <= now.latest:
            future_time = now.latest + 2 * self.truetime.epsilon

        self.change_time = future_time
        return future_time

    def should_apply_change(self, current_time: float) -> bool:
        """
        Check if it's safe to apply the schema change
        Returns True when current_time >= change_time
        """
        if self.change_time is None:
            return False

        # Get current time with uncertainty
        now_interval = self.truetime.now()

        # Apply change only when we're definitely past the change time
        return now_interval.definitely_after(
            TrueTimeInterval(self.change_time, self.change_time)
        )

    def apply_change_safely(self, change_func):
        """
        Wait until safe, then apply change
        """
        if self.change_time is None:
            return

        # Wait until we're definitely past the change time
        self.truetime.after(self.change_time)

        # Now safe to apply change globally
        change_func()

TrueTime Innovations:

Explicit Uncertainty: Time represented as interval, not point
Wait-free Coordination: after() method for synchronization
Global Ordering: Enables globally consistent timestamps
Hardware Redundancy: Multiple atomic clocks and GPS receivers
Software Monitoring: Continuous error bound verification

🌳 4. Interval Tree Clocks: Dynamic Scalability

Interval Tree Clocks (ITC) solve the scaling problem of vector clocks for highly dynamic systems.

from typing import List, Tuple, Optional
import math

class ITCStamp:
    """
    Interval Tree Clock stamp
    Represents causal knowledge as a binary tree

    Properties:
    1. Dynamic: O(log n) size for n events
    2. Fork/Join: Efficient for dynamic process creation
    3. Monotonic: Always increases
    4. Partial Order: Captures happens-before
    """

    def __init__(self):
        # Binary tree representation
        self.tree = self._create_leaf(1)  # Start with single unit

    def _create_leaf(self, value: int):
        return ('leaf', value)

    def _create_node(self, left, right):
        return ('node', left, right)

    def event(self) -> 'ITCStamp':
        """Generate new event (increment stamp)"""
        new_stamp = self._clone()
        new_stamp._increment()
        return new_stamp

    def _increment(self):
        """Internal increment operation"""
        if self.tree[0] == 'leaf':
            value = self.tree[1]
            self.tree = self._create_leaf(value + 1)
        else:
            # node case
            _, left, right = self.tree
            # Try to increment left, if full try right
            left_stamp = ITCStamp()
            left_stamp.tree = left
            right_stamp = ITCStamp()
            right_stamp.tree = right

            if not left_stamp._is_full():
                left_stamp._increment()
                self.tree = self._create_node(left_stamp.tree, right)
            else:
                right_stamp._increment()
                self.tree = self._create_node(left, right_stamp.tree)

    def _is_full(self) -> bool:
        """Check if stamp is at maximum capacity"""
        if self.tree[0] == 'leaf':
            return self.tree[1] >= self._max_leaf_value()
        else:
            _, left, right = self.tree
            left_stamp = ITCStamp()
            left_stamp.tree = left
            right_stamp = ITCStamp()
            right_stamp.tree = right
            return left_stamp._is_full() and right_stamp._is_full()

    def _max_leaf_value(self) -> int:
        """Maximum value for a leaf node"""
        return 2  # For binary trees

    def fork(self) -> Tuple['ITCStamp', 'ITCStamp']:
        """
        Fork into two independent stamps
        Used when creating new processes
        """
        if self.tree[0] == 'leaf':
            value = self.tree[1]
            if value >= 2:
                # Split leaf into two nodes
                left = self._create_leaf(1)
                right = self._create_leaf(value - 1)
                self.tree = self._create_node(left, right)

        # Now tree is a node, split it
        _, left, right = self.tree

        left_stamp = ITCStamp()
        left_stamp.tree = left

        right_stamp = ITCStamp()
        right_stamp.tree = right

        return left_stamp, right_stamp

    def join(self, other: 'ITCStamp') -> 'ITCStamp':
        """
        Join two stamps (merge knowledge)
        Used when processes synchronize
        """
        if self.tree[0] == 'leaf' and other.tree[0] == 'leaf':
            value1 = self.tree[1]
            value2 = other.tree[1]
            return ITCStamp._create_leaf(value1 + value2)
        elif self.tree[0] == 'node' and other.tree[0] == 'node':
            _, left1, right1 = self.tree
            _, left2, right2 = other.tree

            left_stamp1 = ITCStamp()
            left_stamp1.tree = left1
            left_stamp2 = ITCStamp()
            left_stamp2.tree = left2

            right_stamp1 = ITCStamp()
            right_stamp1.tree = right1
            right_stamp2 = ITCStamp()
            right_stamp2.tree = right2

            joined_left = left_stamp1.join(left_stamp2)
            joined_right = right_stamp1.join(right_stamp2)

            result = ITCStamp()
            result.tree = result._create_node(joined_left.tree, joined_right.tree)
            return result
        else:
            # Mismatched types, normalize
            normalized_self = self._normalize()
            normalized_other = other._normalize()
            return normalized_self.join(normalized_other)

    def _normalize(self) -> 'ITCStamp':
        """Convert to normalized form"""
        # Implementation of normalization algorithm
        pass

    def leq(self, other: 'ITCStamp') -> bool:
        """Check if self ≤ other in causal order"""
        if self.tree[0] == 'leaf' and other.tree[0] == 'leaf':
            return self.tree[1] <= other.tree[1]
        elif self.tree[0] == 'node' and other.tree[0] == 'node':
            _, left1, right1 = self.tree
            _, left2, right2 = other.tree

            left_stamp1 = ITCStamp()
            left_stamp1.tree = left1
            left_stamp2 = ITCStamp()
            left_stamp2.tree = left2

            right_stamp1 = ITCStamp()
            right_stamp1.tree = right1
            right_stamp2 = ITCStamp()
            right_stamp2.tree = right2

            return (left_stamp1.leq(left_stamp2) and 
                   right_stamp1.leq(right_stamp2))
        else:
            # Mismatched, normalize first
            return self._normalize().leq(other._normalize())

    def concurrent(self, other: 'ITCStamp') -> bool:
        """Check if stamps are concurrent"""
        return not (self.leq(other) or other.leq(self))

    def _clone(self) -> 'ITCStamp':
        """Create a deep copy"""
        import copy
        new_stamp = ITCStamp()
        new_stamp.tree = copy.deepcopy(self.tree)
        return new_stamp

    def encode(self) -> bytes:
        """Encode stamp to bytes for transmission"""
        # Compact binary encoding
        pass

    @classmethod
    def decode(cls, data: bytes) -> 'ITCStamp':
        """Decode stamp from bytes"""
        pass

# Example: Dynamic microservices with ITC
class MicroserviceOrchestrator:
    """
    Using ITC for causal tracking in dynamic microservices

    Challenge: Services scale up/down dynamically
    Vector clocks would require O(n) space
    ITC provides O(log n) scaling
    """

    def __init__(self):
        self.global_stamp = ITCStamp()
        self.services = {}  # service_id -> ITCStamp
        self.event_log = []

    def create_service(self, service_id: str):
        """Create new service instance"""
        if service_id in self.services:
            raise ValueError(f"Service {service_id} already exists")

        # Fork global knowledge to create new service stamp
        service_stamp, new_global = self.global_stamp.fork()

        self.services[service_id] = service_stamp
        self.global_stamp = new_global

        return service_stamp

    def destroy_service(self, service_id: str):
        """Remove service instance"""
        if service_id not in self.services:
            return

        # Join service knowledge back to global
        service_stamp = self.services[service_id]
        self.global_stamp = self.global_stamp.join(service_stamp)

        del self.services[service_id]

    def service_event(self, service_id: str, event_data: dict) -> ITCStamp:
        """Record event from service"""
        if service_id not in self.services:
            raise ValueError(f"Service {service_id} not found")

        service_stamp = self.services[service_id]

        # Generate new event stamp
        new_stamp = service_stamp.event()
        self.services[service_id] = new_stamp

        # Log event with causal information
        log_entry = {
            'service': service_id,
            'data': event_data,
            'stamp': new_stamp,
            'time': time.time()
        }
        self.event_log.append(log_entry)

        return new_stamp

    def synchronize_services(self, service_a: str, service_b: str):
        """Synchronize two services (e.g., RPC call)"""
        stamp_a = self.services[service_a]
        stamp_b = self.services[service_b]

        # Exchange and merge knowledge
        merged = stamp_a.join(stamp_b)

        # Update both services
        self.services[service_a] = merged
        self.services[service_b] = merged

    def get_causal_history(self, service_id: str) -> List[dict]:
        """Get all events that causally precede service's current state"""
        service_stamp = self.services.get(service_id)
        if not service_stamp:
            return []

        history = []
        for entry in self.event_log:
            if entry['stamp'].leq(service_stamp):
                history.append(entry)

        return history

    def detect_concurrent_events(self, event1: dict, event2: dict) -> bool:
        """Check if two events are concurrent"""
        return event1['stamp'].concurrent(event2['stamp'])

ITC Advantages for Dynamic Systems:

O(log n) Space: Scales better than vector clocks
Dynamic Process Creation: Efficient fork/join operations
Garbage Collection: Built-in through tree normalization
Compact Encoding: Efficient serialization
Partial Order Maintenance: Preserves causal relationships

⚔️ 5. Byzantine Fault Tolerance: Beyond Crash Faults

Byzantine faults include arbitrary, malicious behavior—not just crashes.

import hashlib
import json
from typing import List, Dict, Any, Optional
from dataclasses import dataclass
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes

@dataclass
class ByzantineMessage:
    sender: int
    sequence: int
    payload: Any
    signature: bytes

    def verify(self, public_keys: Dict[int, bytes]) -> bool:
        """Verify message signature"""
        if self.sender not in public_keys:
            return False

        public_key = serialization.load_pem_public_key(
            public_keys[self.sender]
        )

        message_data = json.dumps({
            'sender': self.sender,
            'sequence': self.sequence,
            'payload': self.payload
        }).encode()

        try:
            public_key.verify(
                self.signature,
                message_data,
                padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=padding.PSS.MAX_LENGTH
                ),
                hashes.SHA256()
            )
            return True
        except:
            return False

    def sign(self, private_key: rsa.RSAPrivateKey) -> 'ByzantineMessage':
        """Sign message with private key"""
        message_data = json.dumps({
            'sender': self.sender,
            'sequence': self.sequence,
            'payload': self.payload
        }).encode()

        signature = private_key.sign(
            message_data,
            padding.PSS(
                mgf=padding.MGF1(hashes.SHA256()),
                salt_length=padding.PSS.MAX_LENGTH
            ),
            hashes.SHA256()
        )

        self.signature = signature
        return self

class PracticalByzantineFaultTolerance:
    """
    PBFT: Practical Byzantine Fault Tolerance
    Tolerates f faulty nodes with 3f+1 total nodes
    """

    def __init__(self, node_id: int, total_nodes: int):
        self.node_id = node_id
        self.total_nodes = total_nodes
        self.faulty_limit = (total_nodes - 1) // 3

        # State
        self.view = 0  # Current view number
        self.sequence = 0  # Sequence number for operations
        self.log = []  # Committed operations

        # Cryptographic keys
        self.private_key = rsa.generate_private_key(
            public_exponent=65537,
            key_size=2048
        )

        self.public_key_pem = self.private_key.public_key().public_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PublicFormat.SubjectPublicKeyInfo
        )

        # Known public keys (in real system, from PKI)
        self.public_keys = {}

    def broadcast_prepare(self, operation: Any) -> ByzantineMessage:
        """Phase 1: Primary broadcasts PRE-PREPARE"""
        self.sequence += 1

        message = ByzantineMessage(
            sender=self.node_id,
            sequence=self.sequence,
            payload={
                'type': 'PRE-PREPARE',
                'view': self.view,
                'operation': operation,
                'digest': self._hash_operation(operation)
            },
            signature=b''
        )

        return message.sign(self.private_key)

    def receive_pre_prepare(self, message: ByzantineMessage) -> Optional[ByzantineMessage]:
        """Phase 2: Replicas send PREPARE"""
        if not message.verify(self.public_keys):
            return None

        payload = message.payload
        if payload['type'] != 'PRE-PREPARE':
            return None

        # Verify view and sequence
        if payload['view'] != self.view:
            return None

        # Verify operation digest
        if payload['digest'] != self._hash_operation(payload['operation']):
            return None

        # Send PREPARE message
        prepare_message = ByzantineMessage(
            sender=self.node_id,
            sequence=payload['sequence'],
            payload={
                'type': 'PREPARE',
                'view': self.view,
                'digest': payload['digest']
            },
            signature=b''
        )

        return prepare_message.sign(self.private_key)

    def collect_prepares(self, prepares: List[ByzantineMessage]) -> bool:
        """Phase 3: Check if we have 2f matching PREPARE messages"""
        if len(prepares) < 2 * self.faulty_limit:
            return False

        # Group by digest
        digests = {}
        for prepare in prepares:
            if prepare.verify(self.public_keys):
                digest = prepare.payload['digest']
                digests[digest] = digests.get(digest, 0) + 1

        # Check if any digest has 2f+1 votes
        for digest, count in digests.items():
            if count >= 2 * self.faulty_limit + 1:
                return True

        return False

    def send_commit(self, digest: str) -> ByzantineMessage:
        """Phase 4: Send COMMIT after collecting enough PREPAREs"""
        commit_message = ByzantineMessage(
            sender=self.node_id,
            sequence=self.sequence,
            payload={
                'type': 'COMMIT',
                'view': self.view,
                'digest': digest
            },
            signature=b''
        )

        return commit_message.sign(self.private_key)

    def collect_commits(self, commits: List[ByzantineMessage]) -> Optional[Any]:
        """Phase 5: Execute after collecting 2f+1 COMMITs"""
        if len(commits) < 2 * self.faulty_limit + 1:
            return None

        # Verify all commits are for same digest
        digests = set()
        valid_commits = []

        for commit in commits:
            if commit.verify(self.public_keys):
                digests.add(commit.payload['digest'])
                valid_commits.append(commit)

        if len(digests) != 1 or len(valid_commits) < 2 * self.faulty_limit + 1:
            return None

        # Find the operation with this digest
        digest = list(digests)[0]
        operation = self._find_operation_by_digest(digest)

        if operation:
            # Execute operation
            self.log.append({
                'sequence': self.sequence,
                'operation': operation,
                'digest': digest,
                'view': self.view
            })

            return operation

        return None

    def view_change(self):
        """Handle view change when primary is suspected faulty"""
        self.view += 1

        # New primary is (view mod total_nodes)
        new_primary = self.view % self.total_nodes

        # Collect state from other nodes
        # Send VIEW-CHANGE messages
        # Elect new primary
        # Catch up on missed operations

    def _hash_operation(self, operation: Any) -> str:
        """Create cryptographic hash of operation"""
        operation_str = json.dumps(operation, sort_keys=True).encode()
        return hashlib.sha256(operation_str).hexdigest()

    def _find_operation_by_digest(self, digest: str) -> Optional[Any]:
        """Find operation by its digest"""
        # In real implementation, maintain mapping
        pass

# Example: Byzantine-tolerant blockchain
class ByzantineBlockchain:
    """
    Blockchain with Byzantine fault tolerance
    Each block requires PBFT consensus
    """

    def __init__(self, node_id: int, total_nodes: int):
        self.pbft = PracticalByzantineFaultTolerance(node_id, total_nodes)
        self.chain = []
        self.pending_transactions = []

    def propose_block(self, transactions: List[Any]) -> Optional[Dict]:
        """Propose new block to the network"""
        if self._is_primary():
            block = {
                'index': len(self.chain),
                'previous_hash': self._last_block_hash(),
                'transactions': transactions,
                'timestamp': time.time(),
                'proposer': self.pbft.node_id
            }

            # Start PBFT consensus
            pre_prepare = self.pbft.broadcast_prepare({
                'type': 'BLOCK_PROPOSAL',
                'block': block
            })

            return block

        return None

    def handle_block_proposal(self, message: ByzantineMessage) -> bool:
        """Process block proposal from other node"""
        if not message.verify(self.pbft.public_keys):
            return False

        # Participate in PBFT consensus
        prepare = self.pbft.receive_pre_prepare(message)
        if not prepare:
            return False

        # Collect prepares from other nodes
        # When we have 2f+1 matching prepares, send commit
        # When we have 2f+1 matching commits, execute block

        return True

    def commit_block(self, block: Dict):
        """Add block to chain after consensus"""
        # Verify block structure
        if not self._validate_block(block):
            return

        # Verify consensus proof
        # Add to chain
        self.chain.append(block)

        # Remove transactions from pending
        self.pending_transactions = [
            tx for tx in self.pending_transactions
            if tx not in block['transactions']
        ]

    def _is_primary(self) -> bool:
        """Check if this node is primary for current view"""
        return self.pbft.node_id == (self.pbft.view % self.pbft.total_nodes)

    def _last_block_hash(self) -> str:
        """Get hash of last block in chain"""
        if not self.chain:
            return '0' * 64

        last_block = json.dumps(self.chain[-1], sort_keys=True).encode()
        return hashlib.sha256(last_block).hexdigest()

    def _validate_block(self, block: Dict) -> bool:
        """Validate block structure and transactions"""
        # Implementation depends on application
        return True

Byzantine Fault Models:

Crash Faults: Nodes stop responding
Omission Faults: Nodes drop messages
Timing Faults: Nodes respond too slowly/quickly
Byzantine Faults: Nodes behave arbitrarily (malicious)

PBFT Key Properties:

Safety: All non-faulty nodes execute same operations in same order
Liveness: Eventually execute operations despite faults
Efficiency: 3-phase protocol with O(n²) messages
Optimistic: Normal case requires 1 round trip

📜 6. Paxos Variants: The Consensus Family Tree

Paxos has spawned many variants optimized for different use cases.

from enum import Enum
from typing import List, Dict, Optional, Tuple
import random

class PaxosMessageType(Enum):
    PREPARE = "prepare"
    PROMISE = "promise"
    ACCEPT = "accept"
    ACCEPTED = "accepted"
    NACK = "nack"

class PaxosVariant:
    """Base class for Paxos variants"""

    def __init__(self, node_id: int, quorum_size: int):
        self.node_id = node_id
        self.quorum_size = quorum_size

        # State
        self.proposal_number = (0, node_id)  # (round, node_id)
        self.accepted_value = None
        self.accepted_proposal = None

    def generate_proposal_number(self) -> Tuple[int, int]:
        """Generate unique proposal number"""
        current_round, _ = self.proposal_number
        return (current_round + 1, self.node_id)

    def is_quorum(self, responses: List) -> bool:
        """Check if we have quorum of responses"""
        return len(responses) >= self.quorum_size

class ClassicPaxos(PaxosVariant):
    """Original Multi-Paxos algorithm"""

    def __init__(self, node_id: int, total_nodes: int):
        super().__init__(node_id, total_nodes // 2 + 1)
        self.learned_values = {}
        self.instances = {}  # Consensus instance → value

    def prepare_phase(self, instance_id: int) -> Dict:
        """Phase 1a: Send PREPARE"""
        self.proposal_number = self.generate_proposal_number()

        return {
            'type': PaxosMessageType.PREPARE,
            'instance': instance_id,
            'proposal': self.proposal_number,
            'from': self.node_id
        }

    def handle_promise(self, instance_id: int, promises: List[Dict]) -> Optional[Dict]:
        """Phase 2: Process PROMISEs, send ACCEPT if quorum"""
        if not self.is_quorum(promises):
            return None

        # Find highest accepted value from promises
        highest_accepted = None
        highest_proposal = (0, 0)

        for promise in promises:
            if promise.get('accepted_proposal', (0, 0)) > highest_proposal:
                highest_proposal = promise['accepted_proposal']
                highest_accepted = promise.get('accepted_value')

        # Use highest accepted value or our own
        value = highest_accepted if highest_accepted else self._choose_value(instance_id)

        return {
            'type': PaxosMessageType.ACCEPT,
            'instance': instance_id,
            'proposal': self.proposal_number,
            'value': value,
            'from': self.node_id
        }

    def handle_accepted(self, instance_id: int, accepteds: List[Dict]):
        """Learn value if we have quorum of ACCEPTED"""
        if self.is_quorum(accepteds):
            # All ACCEPTED should have same value
            values = set(msg['value'] for msg in accepteds)
            if len(values) == 1:
                learned_value = values.pop()
                self.learned_values[instance_id] = learned_value
                self.instances[instance_id] = learned_value

    def _choose_value(self, instance_id: int):
        """Choose value to propose"""
        # Application-specific logic
        return f"value-for-{instance_id}"

class FastPaxos(PaxosVariant):
    """
    Fast Paxos: Optimistic fast path
    Uses O(4f+1) nodes instead of O(3f+1)
    """

    def __init__(self, node_id: int, total_nodes: int):
        # Fast Paxos requires larger quorums
        super().__init__(node_id, (3 * total_nodes) // 4 + 1)
        self.fast_quorum = (total_nodes * 3) // 4
        self.classic_quorum = total_nodes // 2 + 1

    def fast_round(self, instance_id: int, value) -> Dict:
        """Fast path: Send value directly to acceptors"""
        return {
            'type': PaxosMessageType.ACCEPT,
            'instance': instance_id,
            'proposal': ('fast', self.node_id),  # Special fast round
            'value': value,
            'from': self.node_id
        }

    def handle_fast_accept(self, instance_id: int, messages: List[Dict]) -> Optional[Dict]:
        """Check if fast path succeeded"""
        # Need fast quorum for fast path
        if len(messages) >= self.fast_quorum:
            # Check for conflict
            values = set(msg['value'] for msg in messages)
            if len(values) == 1:
                # Fast path succeeded
                return {
                    'type': PaxosMessageType.ACCEPTED,
                    'instance': instance_id,
                    'value': values.pop()
                }

        # Fall back to classic Paxos
        return None

class ByzantinePaxos(PaxosVariant):
    """Paxos variant that tolerates Byzantine faults"""

    def __init__(self, node_id: int, total_nodes: int):
        # Byzantine Paxos requires 3f+1 nodes
        self.f = (total_nodes - 1) // 3
        super().__init__(node_id, 2 * self.f + 1)

        # Cryptographic signatures
        self.signatures = {}

    def byzantine_prepare(self, instance_id: int) -> Dict:
        """Phase 1 with signatures"""
        proposal = self.generate_proposal_number()
        signature = self._sign_proposal(instance_id, proposal)

        return {
            'type': PaxosMessageType.PREPARE,
            'instance': instance_id,
            'proposal': proposal,
            'signature': signature,
            'from': self.node_id
        }

    def verify_byzantine_promise(self, promise: Dict) -> bool:
        """Verify promise signature"""
        if 'signature' not in promise:
            return False

        # Verify signature matches sender and content
        # Implementation depends on crypto library
        return True

    def byzantine_accept(self, instance_id: int, value) -> Dict:
        """Phase 2 with signatures"""
        signature = self._sign_value(instance_id, self.proposal_number, value)

        return {
            'type': PaxosMessageType.ACCEPT,
            'instance': instance_id,
            'proposal': self.proposal_number,
            'value': value,
            'signature': signature,
            'from': self.node_id
        }

    def _sign_proposal(self, instance_id: int, proposal: Tuple[int, int]) -> bytes:
        """Sign proposal number"""
        data = f"{instance_id}:{proposal[0]}:{proposal[1]}".encode()
        # Real implementation would use actual signatures
        return hashlib.sha256(data).digest()

    def _sign_value(self, instance_id: int, proposal: Tuple[int, int], value) -> bytes:
        """Sign value"""
        data = f"{instance_id}:{proposal[0]}:{proposal[1]}:{value}".encode()
        return hashlib.sha256(data).digest()

# Example: Paxos-based configuration management
class PaxosConfigurationStore:
    """Configuration store using Paxos for consensus"""

    def __init__(self, node_id: int, nodes: List[int]):
        self.node_id = node_id
        self.nodes = nodes
        self.paxos = ClassicPaxos(node_id, len(nodes))

        # Configuration state
        self.config = {}
        self.config_history = []
        self.config_version = 0

    def propose_config_change(self, key: str, value) -> bool:
        """Propose configuration change"""
        # Create operation
        operation = {
            'type': 'SET_CONFIG',
            'key': key,
            'value': value,
            'version': self.config_version + 1,
            'proposer': self.node_id
        }

        # Run Paxos for this version
        instance_id = self.config_version

        # Phase 1: Prepare
        prepare_msg = self.paxos.prepare_phase(instance_id)
        self._broadcast(prepare_msg)

        # Wait for promises
        # In real implementation, use async/await or callbacks

        # Phase 2: Accept
        # Phase 3: Learn

        # If consensus reached, apply change
        if instance_id in self.paxos.learned_values:
            learned = self.paxos.learned_values[instance_id]
            self._apply_config_operation(learned)
            return True

        return False

    def _apply_config_operation(self, operation: Dict):
        """Apply configuration operation"""
        if operation['type'] == 'SET_CONFIG':
            self.config[operation['key']] = operation['value']
            self.config_version = operation['version']
            self.config_history.append({
                'version': operation['version'],
                'operation': operation,
                'timestamp': time.time()
            })

    def get_config(self, key: str, version: Optional[int] = None):
        """Get configuration value"""
        if version:
            # Get historical value
            for entry in reversed(self.config_history):
                if entry['version'] <= version:
                    if entry['operation']['key'] == key:
                        return entry['operation']['value']
            return None
        else:
            # Get current value
            return self.config.get(key)

    def _broadcast(self, message: Dict):
        """Broadcast message to all nodes"""
        # Implementation depends on network layer
        pass

Paxos Variants Comparison:

Variant	Nodes Required	Message Complexity	Use Case
Classic Paxos	2f+1	O(n²)	General consensus
Fast Paxos	4f+1	O(n) optimistic	Low latency when no conflicts
Byzantine Paxos	3f+1	O(n²) with crypto	Adversarial environments
Cheap Paxos	f+1 (with reconfiguration)	O(n²)	Resource-constrained
Egalitarian Paxos	2f+1	O(n)	Leaderless, symmetric

🚣 7. Raft Extensions: Production Optimizations

Raft's simplicity has led to many production extensions.

from typing import List, Dict, Optional, Set
import time
import threading
from collections import defaultdict

class RaftWithLease:
    """
    Raft with leader leases for linearizable reads
    Leader maintains lease to serve reads without quorum
    """

    def __init__(self, node_id: int, election_timeout: float = 1.0):
        self.node_id = node_id
        self.election_timeout = election_timeout

        # Raft state
        self.state = 'follower'
        self.current_term = 0
        self.voted_for = None
        self.log = []

        # Lease state
        self.lease_expiry = 0
        self.lease_duration = election_timeout / 2

        # For lease validation
        self.last_heartbeat = defaultdict(float)

    def become_leader(self):
        """Transition to leader state"""
        self.state = 'leader'
        self.lease_expiry = time.time() + self.lease_duration

        # Send initial heartbeat to establish lease
        self._broadcast_heartbeat()

    def renew_lease(self):
        """Renew leader lease"""
        if self.state != 'leader':
            return False

        current_time = time.time()

        # Check if majority acknowledge our leadership
        acknowledgments = 0
        for node, last_seen in self.last_heartbeat.items():
            if current_time - last_seen < self.election_timeout:
                acknowledgments += 1

        if acknowledgments >= len(self.last_heartbeat) // 2 + 1:
            self.lease_expiry = current_time + self.lease_duration
            return True

        return False

    def has_valid_lease(self) -> bool:
        """Check if leader lease is still valid"""
        if self.state != 'leader':
            return False

        return time.time() < self.lease_expiry

    def serve_linearizable_read(self, key: str):
        """Serve read with linearizability guarantee"""
        if not self.has_valid_lease():
            # Lease expired, need to go through log
            return self._serve_consistent_read(key)

        # Lease valid, can serve directly
        return self._read_local(key)

    def _serve_consistent_read(self, key: str):
        """Read through log for consistency"""
        # Ensure we're still leader
        if not self._ensure_leadership():
            raise NotLeaderError("Lost leadership")

        # Read local
        return self._read_local(key)

    def _ensure_leadership(self) -> bool:
        """Ensure we're still leader, renew lease if needed"""
        if self.state != 'leader':
            return False

        if not self.has_valid_lease():
            return self.renew_lease()

        return True

    def _read_local(self, key: str):
        """Read from local state machine"""
        # Implementation depends on state machine
        pass

    def _broadcast_heartbeat(self):
        """Send heartbeat to all followers"""
        # Implementation depends on network
        pass

class NotLeaderError(Exception):
    pass

class RaftWithWitness:
    """
    Raft with witness nodes for reduced storage
    Witnesses participate in consensus but don't store log
    """

    def __init__(self, node_id: int, is_witness: bool = False):
        self.node_id = node_id
        self.is_witness = is_witness

        # Regular Raft state
        self.current_term = 0
        self.voted_for = None

        # Witnesses only store metadata
        if is_witness:
            self.log = []  # Empty for witnesses
            self.commit_index = 0
            self.last_applied = 0
        else:
            # Full replica stores complete log
            self.log = []
            self.commit_index = 0
            self.last_applied = 0

        # Track which nodes are witnesses
        self.witnesses = set()

    def can_vote(self) -> bool:
        """Check if node can vote in elections"""
        # Witnesses can vote
        return True

    def can_commit(self) -> bool:
        """Check if node can commit entries"""
        # Only full replicas can commit
        return not self.is_witness

    def receive_entries(self, entries: List[Dict]) -> bool:
        """Receive log entries from leader"""
        if self.is_witness:
            # Witnesses don't store entries
            # Just acknowledge receipt
            return True
        else:
            # Full replicas append to log
            self.log.extend(entries)
            return True

    def get_quorum_size(self, total_nodes: int, witness_count: int) -> int:
        """
        Calculate quorum size with witnesses

        With witnesses, quorum must include:
        1. Majority of all nodes (including witnesses)
        2. At least one full replica
        """
        total = total_nodes + witness_count
        return (total // 2) + 1

    def is_quorum(self, responses: Set[int], 
                  full_replicas: Set[int], 
                  witnesses: Set[int]) -> bool:
        """Check if responses constitute a quorum"""
        total_nodes = len(full_replicas) + len(witnesses)
        quorum_size = self.get_quorum_size(len(full_replicas), len(witnesses))

        if len(responses) < quorum_size:
            return False

        # Check that we have at least one full replica
        if not any(node in full_replicas for node in responses):
            return False

        return True

class RaftWithPreVote:
    """
    Raft with pre-vote to prevent disrupted leaders
    from causing unnecessary elections
    """

    def __init__(self, node_id: int):
        self.node_id = node_id
        self.state = 'follower'
        self.current_term = 0

        # Pre-vote state
        self.pre_vote_term = 0
        self.pre_votes_received = set()

        # Track connectivity
        self.last_contact = defaultdict(float)

    def start_pre_vote(self):
        """Start pre-vote phase before actual election"""
        if self.state != 'candidate':
            return

        self.pre_vote_term = self.current_term + 1
        self.pre_votes_received = {self.node_id}

        # Request pre-votes from other nodes
        self._request_pre_votes()

    def _request_pre_votes(self):
        """Send pre-vote requests to all nodes"""
        message = {
            'type': 'PRE_VOTE_REQUEST',
            'term': self.pre_vote_term,
            'candidate_id': self.node_id,
            'last_log_index': len(self.log) - 1 if self.log else -1,
            'last_log_term': self.log[-1]['term'] if self.log else 0
        }

        self._broadcast(message)

    def handle_pre_vote_request(self, request: Dict) -> Optional[Dict]:
        """Handle incoming pre-vote request"""
        # Check if candidate's log is up-to-date
        if not self._is_candidate_log_up_to_date(request):
            return {
                'type': 'PRE_VOTE_RESPONSE',
                'term': self.current_term,
                'vote_granted': False,
                'from': self.node_id
            }

        # Check if we've recently heard from leader
        current_time = time.time()
        time_since_last_contact = current_time - max(self.last_contact.values(), default=0)

        # Only grant pre-vote if we haven't heard from leader recently
        if time_since_last_contact < self.election_timeout:
            return {
                'type': 'PRE_VOTE_RESPONSE',
                'term': self.current_term,
                'vote_granted': False,
                'from': self.node_id
            }

        # Grant pre-vote
        return {
            'type': 'PRE_VOTE_RESPONSE',
            'term': self.current_term,
            'vote_granted': True,
            'from': self.node_id
        }

    def collect_pre_votes(self, responses: List[Dict]) -> bool:
        """Check if we have majority pre-votes"""
        granted = sum(1 for r in responses if r['vote_granted'])
        total_nodes = self._get_total_nodes()

        if granted >= total_nodes // 2 + 1:
            # Start real election
            return self._start_real_election()

        return False

    def _start_real_election(self) -> bool:
        """Start real election after successful pre-vote"""
        self.current_term = self.pre_vote_term
        self.state = 'candidate'

        # Request actual votes
        return self._request_votes()

    def _is_candidate_log_up_to_date(self, request: Dict) -> bool:
        """Check if candidate's log is at least as up-to-date as ours"""
        our_last_index = len(self.log) - 1 if self.log else -1
        our_last_term = self.log[-1]['term'] if self.log else 0

        cand_last_index = request['last_log_index']
        cand_last_term = request['last_log_term']

        if cand_last_term > our_last_term:
            return True
        elif cand_last_term == our_last_term and cand_last_index >= our_last_index:
            return True

        return False

# Example: Raft-based key-value store with extensions
class ProductionRaftKVStore:
    """Production-ready key-value store using Raft with extensions"""

    def __init__(self, node_id: int, nodes: List[int]):
        self.node_id = node_id
        self.nodes = nodes

        # Use Raft with lease for fast reads
        self.raft = RaftWithLease(node_id)

        # Key-value state machine
        self.kv_store = {}
        self.kv_versions = {}  # key -> [(version, value)]

        # Thread safety
        self.lock = threading.RLock()

    def put(self, key: str, value, require_linearizability: bool = True) -> bool:
        """Put key-value pair with optional linearizability"""
        with self.lock:
            # Create log entry
            entry = {
                'type': 'PUT',
                'key': key,
                'value': value,
                'term': self.raft.current_term,
                'timestamp': time.time()
            }

            # Replicate through Raft
            success = self._replicate_entry(entry)

            if success:
                # Apply to state machine
                self._apply_put(key, value)

                if require_linearizability:
                    # Wait for commitment
                    return self._wait_for_commit(entry)

                return True

            return False

    def get(self, key: str, linearizable: bool = True):
        """Get value with optional linearizability"""
        if linearizable:
            # Use leader lease for linearizable read
            return self.raft.serve_linearizable_read(key)
        else:
            # Eventually consistent read
            with self.lock:
                return self.kv_store.get(key)

    def _apply_put(self, key: str, value):
        """Apply PUT operation to state machine"""
        if key not in self.kv_versions:
            self.kv_versions[key] = []

        version = len(self.kv_versions[key]) + 1
        self.kv_versions[key].append((version, value))
        self.kv_store[key] = value

    def _replicate_entry(self, entry: Dict) -> bool:
        """Replicate log entry through Raft"""
        # Implementation depends on Raft library
        # In production, use libraries like etcd's raft or hashicorp/raft
        pass

    def _wait_for_commit(self, entry: Dict) -> bool:
        """Wait for entry to be committed"""
        # Implementation depends on commit notification
        pass

Raft Extensions Summary:

Extension	Purpose	Use Case
Leader Lease	Fast linearizable reads	Read-heavy workloads
Witness Nodes	Reduce storage requirements	Large clusters with storage constraints
Pre-Vote	Prevent election storms	Unstable networks
Joint Consensus	Membership changes	Rolling updates, scaling
Log Compaction	Control log growth	Long-running systems
Read Index	Leader-less consistent reads	Multi-datacenter

🎯 8. Virtual Synchrony Model: Group Communication

Virtual synchrony provides a consistent view of group membership and message ordering.

from typing import List, Dict, Set, Optional
from enum import Enum
import time
from collections import defaultdict

class VSMessageType(Enum):
    JOIN = "join"
    LEAVE = "leave"
    MULTICAST = "multicast"
    ACK = "ack"
    VIEW_CHANGE = "view_change"

class VirtualSynchrony:
    """
    Virtual Synchrony: Consistent group membership and message ordering

    Properties:
    1. Virtual synchrony: All members see events in same order
    2. View synchrony: All members see membership changes in same order
    3. Safe delivery: Messages delivered only in views where sender present
    """

    def __init__(self, node_id: int):
        self.node_id = node_id

        # Group state
        self.current_view = []  # List of members in current view
        self.view_id = 0

        # Message queues
        self.pending_messages = []  # Messages waiting for view agreement
        self.delivered_messages = []  # Messages already delivered

        # Causal ordering
        self.vector_clock = {}

        # Failure detection
        self.suspected = set()
        self.heartbeats = defaultdict(float)

    def join_group(self, members: List[int]) -> bool:
        """Join or form a new group"""
        if self.node_id in members and self.node_id not in self.current_view:
            # Start view change protocol
            self._initiate_view_change(members)
            return True
        return False

    def leave_group(self):
        """Leave the group gracefully"""
        if self.node_id in self.current_view:
            # Send leave notification
            leave_msg = {
                'type': VSMessageType.LEAVE,
                'sender': self.node_id,
                'view_id': self.view_id,
                'timestamp': time.time()
            }

            self._multicast(leave_msg)

            # Remove from local view
            self.current_view.remove(self.node_id)

    def multicast(self, message: Dict, total_order: bool = True) -> bool:
        """
        Multicast message to group

        total_order: If True, use total ordering (atomic broadcast)
                    If False, use causal ordering
        """
        if self.node_id not in self.current_view:
            return False

        # Create message with metadata
        msg = {
            'type': VSMessageType.MULTICAST,
            'sender': self.node_id,
            'view_id': self.view_id,
            'message_id': self._generate_message_id(),
            'payload': message,
            'timestamp': time.time(),
            'total_order': total_order
        }

        if total_order:
            # For total ordering, use consensus protocol
            return self._total_order_multicast(msg)
        else:
            # For causal ordering, use vector clocks
            return self._causal_multicast(msg)

    def _total_order_multicast(self, message: Dict) -> bool:
        """Atomic broadcast with total ordering"""
        # Phase 1: Propose sequence number
        proposal = {
            'type': 'PROPOSE',
            'message': message,
            'proposer': self.node_id,
            'view': self.view_id
        }

        # Send to all members
        responses = self._send_to_all(proposal)

        # Phase 2: Agree on sequence (consensus)
        if self._is_quorum(responses):
            # Assign sequence number
            sequence = self._assign_sequence(message)

            # Phase 3: Deliver in order
            self._deliver_in_order(message, sequence)
            return True

        return False

    def _causal_multicast(self, message: Dict) -> bool:
        """Multicast with causal ordering"""
        # Update vector clock
        if self.node_id not in self.vector_clock:
            self.vector_clock[self.node_id] = 0
        self.vector_clock[self.node_id] += 1

        # Attach vector clock to message
        message['vector_clock'] = self.vector_clock.copy()

        # Broadcast message
        self._broadcast(message)

        # Buffer for causal delivery
        self.pending_messages.append(message)
        self._deliver_causally()

        return True

    def _deliver_causally(self):
        """Deliver messages in causal order"""
        # Sort pending messages by vector clock
        deliverable = []

        for msg in self.pending_messages:
            if self._causally_ready(msg):
                deliverable.append(msg)

        # Sort by timestamp (or other deterministic order)
        deliverable.sort(key=lambda x: (x['timestamp'], x['sender']))

        for msg in deliverable:
            self._deliver_message(msg)
            self.pending_messages.remove(msg)

    def _causally_ready(self, message: Dict) -> bool:
        """Check if message is causally ready for delivery"""
        if 'vector_clock' not in message:
            return True

        msg_vc = message['vector_clock']

        # Check if all causal dependencies are satisfied
        for node, counter in msg_vc.items():
            if self.vector_clock.get(node, 0) < counter:
                # Missing causal dependency
                return False

        return True

    def _deliver_message(self, message: Dict):
        """Deliver message to application"""
        # Update vector clock
        if 'vector_clock' in message:
            for node, counter in message['vector_clock'].items():
                self.vector_clock[node] = max(self.vector_clock.get(node, 0), counter)

        # Increment own counter for delivery event
        self.vector_clock[self.node_id] = self.vector_clock.get(self.node_id, 0) + 1

        # Add to delivered messages
        self.delivered_messages.append(message)

        # Notify application
        self._application_deliver(message['payload'])

    def _initiate_view_change(self, new_members: List[int]):
        """Initiate view change protocol"""
        # Phase 1: Propose new view
        view_proposal = {
            'type': VSMessageType.VIEW_CHANGE,
            'proposed_view': new_members,
            'view_id': self.view_id + 1,
            'proposer': self.node_id
        }

        # Send to all proposed members
        responses = []
        for member in new_members:
            if member != self.node_id:
                # In real system, send over network
                pass

        # Phase 2: Wait for acknowledgments
        if len(responses) >= len(new_members) // 2 + 1:
            # Phase 3: Install new view
            self._install_view(new_members, self.view_id + 1)

    def _install_view(self, members: List[int], view_id: int):
        """Install new view"""
        old_view = set(self.current_view)
        new_view = set(members)

        # Members that left
        left = old_view - new_view

        # Members that joined
        joined = new_view - old_view

        # Update state
        self.current_view = members.copy()
        self.view_id = view_id

        # Handle state transfer for new members
        for new_member in joined:
            self._transfer_state_to(new_member)

        # Clean up state for departed members
        for departed in left:
            self._cleanup_state_for(departed)

    def _transfer_state_to(self, new_member: int):
        """Transfer state to new group member"""
        # Send current view
        # Send undelivered messages
        # Send vector clock state
        pass

    def _application_deliver(self, message):
        """Callback for application to process delivered message"""
        # Override in subclass
        pass

    def _generate_message_id(self) -> str:
        """Generate unique message ID"""
        return f"{self.node_id}:{time.time():.6f}:{hash(str(self.delivered_messages))}"

    def _is_quorum(self, responses: List) -> bool:
        """Check if we have quorum of responses"""
        return len(responses) >= len(self.current_view) // 2 + 1

    def _send_to_all(self, message: Dict) -> List[Dict]:
        """Send message to all group members"""
        # Implementation depends on network
        return []

    def _broadcast(self, message: Dict):
        """Broadcast message to group"""
        # Implementation depends on network
        pass

# Example: Virtual synchrony for distributed gaming
class DistributedGameServer:
    """Multiplayer game server using virtual synchrony"""

    def __init__(self, server_id: int):
        self.vs = VirtualSynchrony(server_id)
        self.game_state = {}
        self.players = {}

        # Game-specific
        self.game_tick = 0
        self.pending_actions = []

    def player_join(self, player_id: int, player_data: Dict):
        """Handle player joining game"""
        # Update local state
        self.players[player_id] = player_data

        # Multicast join event
        join_msg = {
            'action': 'PLAYER_JOIN',
            'player_id': player_id,
            'player_data': player_data,
            'tick': self.game_tick
        }

        self.vs.multicast(join_msg, total_order=True)

    def player_action(self, player_id: int, action: Dict):
        """Handle player action"""
        # Validate action
        if not self._validate_action(player_id, action):
            return False

        # Create action message
        action_msg = {
            'action': 'PLAYER_ACTION',
            'player_id': player_id,
            'action': action,
            'tick': self.game_tick,
            'timestamp': time.time()
        }

        # Use causal ordering for game actions
        return self.vs.multicast(action_msg, total_order=False)

    def game_tick_update(self):
        """Process game tick"""
        self.game_tick += 1

        # Collect all actions for this tick
        tick_actions = [a for a in self.pending_actions 
                       if a.get('tick') == self.game_tick - 1]

        # Apply actions in deterministic order
        tick_actions.sort(key=lambda x: (x['timestamp'], x['player_id']))

        for action in tick_actions:
            self._apply_game_action(action)

        # Broadcast game state update
        state_update = {
            'action': 'GAME_STATE_UPDATE',
            'tick': self.game_tick,
            'game_state': self.game_state,
            'players': self.players
        }

        self.vs.multicast(state_update, total_order=True)

    def _application_deliver(self, message: Dict):
        """Process delivered messages"""
        action = message.get('action')

        if action == 'PLAYER_JOIN':
            self._handle_player_join(message)
        elif action == 'PLAYER_ACTION':
            self._handle_player_action(message)
        elif action == 'GAME_STATE_UPDATE':
            self._handle_state_update(message)
        elif action == 'PLAYER_LEAVE':
            self._handle_player_leave(message)

    def _handle_player_join(self, message: Dict):
        """Handle player join event"""
        player_id = message['player_id']
        player_data = message['player_data']

        # Update local state
        self.players[player_id] = player_data

        # Initialize player in game state
        self.game_state[player_id] = {
            'position': (0, 0, 0),
            'health': 100,
            'score': 0
        }

    def _handle_player_action(self, message: Dict):
        """Handle player action event"""
        # Add to pending actions for next tick
        self.pending_actions.append(message)

    def _apply_game_action(self, action: Dict):
        """Apply game action to state"""
        player_id = action['player_id']
        action_data = action['action']

        # Update game state based on action
        # This is game-specific logic
        pass

    def _handle_state_update(self, message: Dict):
        """Handle game state update"""
        # Update local state
        self.game_state = message['game_state']
        self.players = message['players']
        self.game_tick = message['tick']

    def _validate_action(self, player_id: int, action: Dict) -> bool:
        """Validate player action"""
        if player_id not in self.players:
            return False

        # Game-specific validation
        return True

Virtual Synchrony Guarantees:

View Synchrony: All members see same sequence of views
Message Ordering: Messages delivered in same order to all members
Atomicity: All or nothing delivery within a view
Virtual Synchrony: Messages from same view delivered in same relative order

📊 9. Distributed Complexity Theory

Understanding the fundamental limits of distributed computation.

from typing import List, Set, Dict, Tuple
import math
from enum import Enum

class ComplexityClass(Enum):
    """Complexity classes for distributed problems"""
    LOCAL = "O(1) rounds"
    LOGARITHMIC = "O(log n) rounds"
    POLYLOG = "O(polylog n) rounds"
    POLYNOMIAL = "O(poly n) rounds"
    GLOBAL = "Ω(n) rounds"

class DistributedProblem:
    """Base class for distributed problems"""

    def __init__(self, n: int):
        self.n = n  # Number of nodes
        self.rounds = 0
        self.messages_sent = 0
        self.bits_sent = 0

    def lower_bound_rounds(self) -> int:
        """Lower bound on number of communication rounds"""
        raise NotImplementedError

    def lower_bound_messages(self) -> int:
        """Lower bound on number of messages"""
        raise NotImplementedError

    def lower_bound_bits(self) -> int:
        """Lower bound on number of bits"""
        raise NotImplementedError

    def complexity_class(self) -> ComplexityClass:
        """Complexity class of the problem"""
        raise NotImplementedError

class LeaderElection(DistributedProblem):
    """Leader election in anonymous ring"""

    def __init__(self, n: int):
        super().__init__(n)

    def lower_bound_rounds(self) -> int:
        """
        Lower bound: Ω(n) rounds for anonymous ring
        Even with unique IDs: Ω(log n) rounds
        """
        return self.n  # For anonymous ring

    def lower_bound_messages(self) -> int:
        """
        Lower bound: Ω(n log n) messages for comparison-based
        With unique IDs: O(n) messages possible
        """
        return self.n * math.ceil(math.log2(self.n))

    def lower_bound_bits(self) -> int:
        """Each message needs Ω(log n) bits for ID"""
        return self.lower_bound_messages() * math.ceil(math.log2(self.n))

    def complexity_class(self) -> ComplexityClass:
        return ComplexityClass.GLOBAL

class ConsensusProblem(DistributedProblem):
    """Distributed consensus with crash faults"""

    def __init__(self, n: int, f: int):
        super().__init__(n)
        self.f = f  # Maximum faulty nodes

    def lower_bound_rounds(self) -> int:
        """
        FLP: No solution in asynchronous systems
        Synchronous: f+1 rounds necessary and sufficient
        """
        return self.f + 1

    def lower_bound_messages(self) -> int:
        """
        Lower bound: Ω(n²) messages for some solutions
        Optimal: O(n) messages per round
        """
        return (self.f + 1) * self.n

    def lower_bound_bits(self) -> int:
        """Each message contains value + metadata"""
        return self.lower_bound_messages() * 32  # Assuming 32-bit values

    def complexity_class(self) -> ComplexityClass:
        return ComplexityClass.POLYNOMIAL

class ColoringProblem(DistributedProblem):
    """Graph coloring in distributed setting"""

    def __init__(self, n: int, delta: int):
        super().__init__(n)
        self.delta = delta  # Maximum degree

    def lower_bound_rounds(self) -> int:
        """
        Δ+1 coloring:
        - Deterministic: Ω(log* n) rounds
        - Randomized: O(log log n) rounds expected
        """
        return math.ceil(math.log2(self.delta))

    def lower_bound_messages(self) -> int:
        """Each node communicates with neighbors each round"""
        return self.n * self.delta * self.lower_bound_rounds()

    def complexity_class(self) -> ComplexityClass:
        return ComplexityClass.LOGARITHMIC

class MSTProblem(DistributedProblem):
    """Minimum Spanning Tree in distributed setting"""

    def __init__(self, n: int):
        super().__init__(n)

    def lower_bound_rounds(self) -> int:
        """
        Lower bound: Ω(√(n/log n) + D) rounds
        where D is network diameter
        Best known: O(√n log* n) rounds
        """
        return math.ceil(math.sqrt(self.n / math.log(self.n)))

    def lower_bound_messages(self) -> int:
        """Ω(m) messages where m is number of edges"""
        # Complete graph has m = n(n-1)/2 edges
        return self.n * (self.n - 1) // 2

    def complexity_class(self) -> ComplexityClass:
        return ComplexityClass.POLYNOMIAL

class LowerBoundProver:
    """Prove lower bounds using information theory"""

    @staticmethod
    def fooling_set_method(problem_size: int, output_size: int) -> int:
        """
        Fooling set method for communication complexity

        Returns: Lower bound on bits that must be communicated
        """
        # Size of fooling set gives lower bound
        return math.ceil(math.log2(problem_size / output_size))

    @staticmethod
    def crossing_sequence_method(diameter: int, states: int) -> int:
        """
        Crossing sequence method for round complexity

        Returns: Lower bound on number of rounds
        """
        # Number of different crossing sequences
        sequences = states ** diameter
        return math.ceil(math.log2(sequences))

    @staticmethod
    def indistinguishability_argument(configurations: int) -> int:
        """
        Indistinguishability argument for impossibility proofs

        Used in FLP impossibility proof
        """
        return math.ceil(math.log2(configurations))

# Example: Proving lower bounds for specific problems
class LowerBoundExamples:
    """Concrete lower bound proofs"""

    @staticmethod
    def prove_consensus_lower_bound(n: int, f: int):
        """Proof sketch for consensus lower bounds"""
        print(f"=== Consensus Lower Bounds (n={n}, f={f}) ===")

        # Round lower bound: f+1
        print(f"1. Round lower bound: {f+1} rounds")
        print("   Proof: Need f+1 rounds to wait for all non-faulty nodes")

        # Message lower bound
        min_messages = (f + 1) * n
        print(f"2. Message lower bound: Ω({min_messages}) messages")
        print("   Proof: Each round requires at least n messages")

        # Impossibility in async
        print("3. FLP Impossibility: No deterministic solution in async systems")
        print("   Proof: Always possible to keep system in bivalent state")

    @staticmethod
    def prove_leader_election_lower_bound(n: int):
        """Proof sketch for leader election lower bounds"""
        print(f"\n=== Leader Election Lower Bounds (n={n}) ===")

        # For anonymous ring
        print(f"1. Anonymous ring: Ω({n}) rounds")
        print("   Proof: Need to break symmetry, information travels O(1) per round")

        # Message complexity
        print(f"2. Message complexity: Ω({n} log {n}) messages")
        print("   Proof: Comparison-based algorithm needs pairwise comparisons")

        # With unique IDs
        print(f"3. With unique IDs: O({n}) messages possible")
        print("   Proof: Can use ID propagation algorithms")

# Complexity hierarchy of distributed problems
distributed_hierarchy = {
    ComplexityClass.LOCAL: [
        "Maximal Independent Set (randomized)",
        "Weak coloring",
        "Network decomposition (fragmented)"
    ],
    ComplexityClass.LOGARITHMIC: [
        "MIS (deterministic)",
        "Δ+1 coloring",
        "Maximal matching"
    ],
    ComplexityClass.POLYLOG: [
        "Minimum dominating set",
        "Minimum vertex cover approximation",
        "Network decomposition"
    ],
    ComplexityClass.POLYNOMIAL: [
        "Minimum spanning tree",
        "Shortest paths",
        "Max flow"
    ],
    ComplexityClass.GLOBAL: [
        "Consensus",
        "Byzantine agreement",
        "Leader election (anonymous)"
    ]
}

Key Complexity Results:

LOCAL Model: Problems solvable in O(1) rounds
CONGEST Model: Limited bandwidth per edge
CONGESTED CLIQUE: All pairs can communicate directly
Beeping Model: Extremely limited communication

⚖️ 10. Lower Bounds: The Minimum Cost of Coordination

Proving what's fundamentally impossible or expensive.

import numpy as np
from typing import List, Dict, Set
import math

class InformationTheoreticLowerBounds:
    """Lower bounds using information theory"""

    @staticmethod
    def communication_complexity(input_size: int, output_size: int) -> int:
        """
        Lower bound on bits that must be communicated

        Based on: Need to distinguish between all possible inputs
        """
        # Number of possible inputs
        possible_inputs = 2 ** input_size

        # Number of possible outputs
        possible_outputs = 2 ** output_size

        # Minimum bits to distinguish inputs
        return math.ceil(math.log2(possible_inputs / possible_outputs))

    @staticmethod
    def round_complexity(diameter: int, local_states: int) -> int:
        """
        Lower bound on number of rounds

        Based on: Information propagates at most one hop per round
        """
        # Information needs to travel diameter distance
        return diameter

    @staticmethod
    def message_complexity(nodes: int, edges: int) -> int:
        """
        Lower bound on number of messages

        Based on: Need to communicate across cuts in network
        """
        # Minimum cut in network
        return edges // 2

    @staticmethod
    def storage_complexity(state_size: int, replicas: int) -> int:
        """
        Lower bound on storage overhead

        Based on: Need to store enough information to recover from failures
        """
        # For f faults, need f+1 replicas
        return state_size * replicas

class ConsensusLowerBounds:
    """Specific lower bounds for consensus problems"""

    @staticmethod
    def flp_impossibility():
        """
        Fischer-Lynch-Paterson Impossibility

        No deterministic consensus protocol is possible in 
        asynchronous systems with one crash fault
        """
        proof_steps = [
            "1. Assume a protocol exists that always terminates",
            "2. Construct an execution that keeps system in bivalent state",
            "3. Show adversary can always delay messages to maintain bivalence",
            "4. Therefore, protocol cannot always terminate → contradiction"
        ]
        return proof_steps

    @staticmethod
    def dls_impossibility(partial_synchrony: bool):
        """
        Dwork-Lynch-Stockmeyer Impossibility

        In partially synchronous systems:
        - If clocks can drift unboundedly, consensus impossible
        - With bounded drift, consensus possible with failure detectors
        """
        if partial_synchrony:
            return "Possible with failure detectors (e.g., ◊P)"
        else:
            return "Impossible with unbounded clock drift"

    @staticmethod
    def byzantine_lower_bound(nodes: int, faults: int) -> Dict:
        """
        Lower bounds for Byzantine consensus

        Requires: n > 3f for authentication
                 n > 2f for no authentication
        """
        results = {
            'minimum_nodes_with_auth': 3 * faults + 1,
            'minimum_nodes_no_auth': 2 * faults + 1,
            'round_lower_bound': faults + 1,
            'message_complexity': 'Ω(n²)',
            'cryptographic_assumptions': 'Needed for digital signatures'
        }
        return results

class DataStructureLowerBounds:
    """Lower bounds for distributed data structures"""

    @staticmethod
    def crdt_convergence(state_size: int, operations: int) -> int:
        """
        Lower bound on metadata for CRDT convergence

        Need to track enough information to merge any pair of states
        """
        # For grow-only counter: O(n) where n = number of replicas
        # For observed-remove set: O(m) where m = number of elements
        return state_size * math.log2(operations)

    @staticmethod
    def consistent_hashing_load(distribution: str, nodes: int, items: int) -> float:
        """
        Load balance lower bounds for consistent hashing

        Returns: Minimum maximum load any node must handle
        """
        if distribution == 'uniform':
            # With k choices: O(log log n / log k) improvement
            base_load = items / nodes
            optimal_load = base_load * (1 + 1/math.log2(nodes))
            return optimal_load
        elif distribution == 'bounded':
            # With bounded loads: O(1) maximum load
            return math.ceil(items / nodes)

    @staticmethod
    def quorum_system_availability(quorum_size: int, 
                                  failure_prob: float,
                                  nodes: int) -> float:
        """
        Lower bound on availability of quorum systems

        Trade-off between load and availability
        """
        # Probability all quorums fail
        prob_all_fail = failure_prob ** quorum_size

        # Availability lower bound
        availability = 1 - prob_all_fail

        # Lower bound from probabilistic analysis
        lower_bound = 1 - math.exp(-nodes * failure_prob)

        return max(availability, lower_bound)

class NetworkLowerBounds:
    """Lower bounds for network protocols"""

    @staticmethod
    def tcp_throughput(rtt: float, loss_rate: float, mss: int = 1460) -> float:
        """
        Mathis Equation: TCP throughput upper bound

        throughput <= (MSS / RTT) * (1 / sqrt(p))
        where p = loss rate
        """
        if loss_rate == 0:
            return float('inf')

        throughput = (mss / rtt) * (1 / math.sqrt(loss_rate))
        return throughput

    @staticmethod
    def bandwidth_delay_product(bandwidth: float, rtt: float) -> float:
        """
        BDP = Bandwidth × Round Trip Time

        Minimum buffer size needed to fill the pipe
        """
        return bandwidth * rtt

    @staticmethod
    def queueing_delay(arrival_rate: float, 
                      service_rate: float,
                      cv_a: float = 1.0,
                      cv_s: float = 1.0) -> float:
        """
        Kingman's Formula for G/G/1 queue

        E[W] ≈ (ρ / (1 - ρ)) * ((cv_a² + cv_s²) / 2) * E[S]
        where ρ = λ/μ, cv = coefficient of variation
        """
        if arrival_rate >= service_rate:
            return float('inf')

        rho = arrival_rate / service_rate
        service_time = 1 / service_rate

        wait_time = (rho / (1 - rho)) * ((cv_a**2 + cv_s**2) / 2) * service_time
        return wait_time

# Example: Proving specific lower bounds
class ConcreteLowerBoundProofs:
    """Concrete proofs of distributed systems lower bounds"""

    @staticmethod
    def proof_consensus_messages(n: int):
        """
        Proof that consensus requires Ω(n²) messages
        in worst case for some algorithms
        """
        print("=== Proof: Consensus requires Ω(n²) messages ===")
        print("\n1. Setup:")
        print(f"   - n = {n} nodes")
        print("   - Each node has initial value 0 or 1")
        print("   - Need all nodes to decide same value")

        print("\n2. Adversarial strategy:")
        print("   - Network is complete graph")
        print("   - Adversary controls scheduling")
        print("   - Can delay specific messages")

        print("\n3. Information-theoretic argument:")
        print("   - Each node initially knows only its own value")
        print("   - To learn others' values, must receive messages")
        print(f"   - Need at least n(n-1)/2 messages in worst case")

        print("\n4. Reduction:")
        print("   - Reduce from set disjointness problem")
        print("   - Known to require Ω(n) bits for 2-party")
        print("   - Generalizes to Ω(n²) for n-party")

        print(f"\n5. Result: Ω({n}²) messages required")

    @staticmethod
    def proof_leader_election_anonymous(n: int):
        """
        Proof that leader election in anonymous ring
        requires Ω(n) rounds
        """
        print(f"\n=== Proof: Leader election requires Ω({n}) rounds in anonymous ring ===")

        print("\n1. Symmetry argument:")
        print("   - All nodes identical (anonymous)")
        print("   - Same program, no unique IDs")
        print("   - Initially symmetric situation")

        print("\n2. Information propagation:")
        print("   - Ring diameter = ⌊n/2⌋")
        print("   - Information travels 1 hop per round")
        print(f"   - Need at least ⌊{n}/2⌋ rounds to break symmetry")

        print("\n3. Indistinguishability:")
        print("   - After k rounds, nodes at distance > k see same neighborhood")
        print("   - Cannot distinguish themselves")
        print("   - Need enough rounds for information to travel full diameter")

        print(f"\n4. Result: Ω({n}) rounds required")

    @staticmethod
    def proof_byzantine_agreement(f: int):
        """
        Proof that Byzantine agreement requires n > 3f
        """
        print(f"\n=== Proof: Byzantine agreement requires n > 3{f} ===")

        print("\n1. Setup:")
        print(f"   - f = {f} Byzantine (malicious) nodes")
        print("   - Rest are honest")
        print("   - Need all honest nodes to agree")

        print("\n2. Partition argument:")
        print("   - Divide nodes into 3 groups of size f")
        print("   - Byzantine nodes can lie to different groups")
        print("   - If n ≤ 3f, Byzantine can prevent agreement")

        print("\n3. Contradiction scenario:")
        print("   - Group A sees value 0 from Byzantine")
        print("   - Group B sees value 1 from Byzantine")
        print("   - Group C (Byzantine) can be inconsistent")
        print("   - Honest nodes in A and B cannot agree")

        print(f"\n4. Result: Need n > 3{f} for Byzantine agreement")

Fundamental Lower Bounds Summary:

FLP Impossibility: No async consensus with crash faults
CAP Theorem: Can't have all three properties
Consensus Lower Bound: f+1 rounds with f crash faults
Byzantine Agreement: Need n > 3f nodes
Leader Election: Ω(n) rounds in anonymous networks
Communication Complexity: Ω(n log n) bits for sorting

🚫 11. Impossibility Results: What Can't Be Done

Some problems are fundamentally impossible in distributed systems.

from typing import Set, List, Dict
import itertools

class ImpossibilityResults:
    """Collection of impossibility results in distributed computing"""

    @staticmethod
    def flp_impossibility_detail():
        """
        Detailed proof of FLP impossibility

        Theorem: No deterministic consensus protocol can guarantee
        termination in an asynchronous system with one crash fault
        """
        proof = {
            'assumptions': [
                "Asynchronous message passing",
                "Crash failures (stop failures)",
                "Deterministic algorithms",
                "No message loss (fair loss)",
                "Reliable links"
            ],
            'definitions': [
                "Configuration: Global state of system",
                "Step: Action by a single process",
                "Schedule: Sequence of steps",
                "Univalent: All extensions lead to same decision",
                "Bivalent: Some extensions lead to 0, some to 1"
            ],
            'proof_steps': [
                "1. Show initial configuration is bivalent",
                "2. Show we can keep system bivalent forever",
                "3. Construct adversary that delays messages to maintain bivalence",
                "4. Therefore, cannot guarantee termination"
            ],
            'implications': [
                "Need randomization for guaranteed termination",
                "Or need partial synchrony assumptions",
                "Or need failure detectors"
            ]
        }
        return proof

    @staticmethod
    def cap_impossibility_detail():
        """
        CAP Theorem impossibility

        Theorem: Can't achieve all three of:
        - Consistency (all nodes see same data)
        - Availability (every request gets response)
        - Partition tolerance (system works despite network partitions)
        """
        proof = {
            'formal_statement': "In async network with partitions, "
                              "impossible to implement read/write register "
                              "that is both available and atomic",
            'proof_sketch': [
                "1. Assume partition separates nodes",
                "2. Write happens on one side",
                "3. Read happens on other side",
                "4. To be available, must respond",
                "5. To be consistent, must return latest write",
                "6. Impossible without communication across partition",
                "7. Contradiction"
            ],
            'practical_implications': [
                "Must choose AP or CP during partitions",
                "Can have CA only if no partitions guaranteed",
                "Real systems choose based on use case"
            ]
        }
        return proof

    @staticmethod
    def consensus_impossibility(models: List[str]) -> Dict[str, bool]:
        """
        Check consensus impossibility under different models
        """
        results = {}

        for model in models:
            if model == 'async_crash':
                results[model] = 'impossible (FLP)'
            elif model == 'async_byzantine':
                results[model] = 'impossible'
            elif model == 'partial_sync_crash':
                results[model] = 'possible with failure detectors'
            elif model == 'sync_crash':
                results[model] = 'possible (f+1 rounds)'
            elif model == 'sync_byzantine':
                results[model] = f'possible (n > 3f)'
            elif model == 'anonymous':
                results[model] = 'impossible for some problems'

        return results

    @staticmethod
    def two_generals_problem():
        """
        Two Generals Problem (coordinated attack)

        Theorem: Impossible to guarantee coordinated attack
        over unreliable communication
        """
        return {
            'problem': "Two generals need to coordinate attack time",
            'constraints': [
                "Communication via messengers",
                "Messengers can be captured",
                "Need agreement on exact time",
                "Attack fails without perfect coordination"
            ],
            'impossibility': "No protocol guarantees agreement with probability 1",
            'reason': "Need infinite message exchanges for certainty",
            'solution': "Accept probabilistic agreement or use trusted third party"
        }

    @staticmethod
    def byzantine_generals_problem(f: int):
        """
        Byzantine Generals Problem

        Theorem: Need n > 3f for Byzantine agreement
        with oral messages (no signatures)
        """
        return {
            'problem': f"{f} traitorous generals among n total",
            'requirements': [
                "All loyal generals decide same plan",
                "If commander loyal, all loyal generals follow his plan"
            ],
            'impossibility': f"If n ≤ 3f, no solution exists",
            'proof': "Partition generals into 3 groups, traitors can confuse",
            'with_signatures': f"With signatures, need n > 2f"
        }

    @staticmethod
    def deterministic_consensus_async():
        """
        Strengthening of FLP

        Theorem: No deterministic protocol solves consensus
        in async systems even with:
        - Only one crash fault
        - Reliable links
        - No message loss
        """
        return {
            'stronger_form': "Even weaker than termination impossible",
            'results': [
                "Cannot guarantee agreement always",
                "Cannot guarantee validity always",
                "Termination is the impossible property"
            ],
            'workarounds': [
                "Randomized algorithms",
                "Failure detectors (◊P)",
                "Partial synchrony assumptions",
                "Weaker consistency models"
            ]
        }

class ImpossibilityDemonstrator:
    """Demonstrate impossibility through simulation"""

    def __init__(self):
        self.configurations = set()
        self.schedules = []

    def demonstrate_flp(self, processes: int):
        """Demonstrate FLP through state exploration"""
        print(f"\n=== Demonstrating FLP for {processes} processes ===")

        # Initial states
        initial_states = self._generate_initial_states(processes)
        print(f"Initial configurations: {len(initial_states)}")

        # Explore reachable states
        reachable = self._explore_reachable(initial_states)

        # Check for bivalent configurations
        bivalent = self._find_bivalent(reachable)

        print(f"Bivalent configurations found: {len(bivalent) > 0}")

        if bivalent:
            print("\nFound bivalent configuration that can be kept bivalent:")
            print(self._format_configuration(next(iter(bivalent))))

    def _generate_initial_states(self, processes: int) -> Set:
        """Generate all possible initial states"""
        # Each process has value 0 or 1
        states = set()
        for values in itertools.product([0, 1], repeat=processes):
            states.add(tuple(values))
        return states

    def _explore_reachable(self, initial_states: Set) -> Set:
        """Explore reachable configurations"""
        visited = set()
        stack = list(initial_states)

        while stack:
            config = stack.pop()
            if config in visited:
                continue

            visited.add(config)

            # Generate next configurations
            # Simulate process steps
            for next_config in self._next_configurations(config):
                if next_config not in visited:
                    stack.append(next_config)

        return visited

    def _next_configurations(self, config: tuple) -> List[tuple]:
        """Generate possible next configurations"""
        next_configs = []
        n = len(config)

        # Simulate each process taking a step
        for i in range(n):
            # Process i crashes (stops)
            crashed_config = config[:i] + ('X',) + config[i+1:]
            next_configs.append(crashed_config)

            # Process i takes step (changes value)
            new_value = 1 - config[i] if config[i] in [0, 1] else config[i]
            stepped_config = config[:i] + (new_value,) + config[i+1:]
            next_configs.append(stepped_config)

        return next_configs

    def _find_bivalent(self, configurations: Set) -> Set:
        """Find bivalent configurations"""
        bivalent = set()

        for config in configurations:
            reachable_decisions = self._reachable_decisions(config)
            if 0 in reachable_decisions and 1 in reachable_decisions:
                bivalent.add(config)

        return bivalent

    def _reachable_decisions(self, config: tuple) -> Set:
        """Find all decisions reachable from configuration"""
        decisions = set()

        # Simplified: decision is when all non-crashed agree
        if all(v == 0 or v == 'X' for v in config):
            decisions.add(0)
        if all(v == 1 or v == 'X' for v in config):
            decisions.add(1)

        return decisions

    def _format_configuration(self, config: tuple) -> str:
        """Format configuration for display"""
        return f"[{', '.join(str(v) for v in config)}]"

# Example: Understanding impossibility in practice
class PracticalImpossibilities:
    """Real-world implications of impossibility results"""

    @staticmethod
    def database_implications():
        """How impossibility affects database design"""
        implications = {
            'ACID_transactions': [
                "Isolation levels trade off consistency for performance",
                "Serializable isolation expensive, often not used",
                "Snapshot isolation common compromise"
            ],
            'distributed_transactions': [
                "Two-phase commit blocks on coordinator failure",
                "Three-phase commit avoids blocking but more complex",
                "Paxos/Raft used for consensus instead"
            ],
            'replication': [
                "Synchronous replication: consistent but slow",
                "Asynchronous replication: fast but can lose data",
                "Semi-synchronous: compromise"
            ]
        }
        return implications

    @staticmethod
    def system_design_choices():
        """Design choices forced by impossibility"""
        choices = [
            {
                'problem': "Leader election",
                'impossibility': "Anonymous networks require Ω(n) time",
                'solution': "Use unique IDs or random delays"
            },
            {
                'problem': "Consensus",
                'impossibility': "Async + crash faults = no termination guarantee",
                'solution': "Use failure detectors or randomization"
            },
            {
                'problem': "Byzantine fault tolerance",
                'impossibility': "Need n > 3f for agreement",
                'solution': "Use larger clusters or trusted components"
            },
            {
                'problem': "Coordinated attack",
                'impossibility': "Need infinite messages for certainty",
                'solution': "Accept probabilistic guarantees"
            }
        ]
        return choices

    @staticmethod
    def working_around_impossibility():
        """Practical workarounds for theoretical impossibilities"""
        workarounds = {
            'flp': [
                "Use randomized algorithms (e.g., Ben-Or's algorithm)",
                "Use failure detectors (e.g., ◊P for Paxos)",
                "Assume partial synchrony (most real systems do)",
                "Accept occasional non-termination (with timeouts)"
            ],
            'cap': [
                "Choose AP or CP based on application needs",
                "Use tunable consistency (e.g., Cassandra quorums)",
                "Separate data into different consistency tiers",
                "Use conflict resolution for AP systems"
            ],
            'byzantine': [
                "Use cryptographic signatures",
                "Increase replication factor (n > 3f)",
                "Use trusted hardware (SGX, TPM)",
                "Economic incentives (blockchain-style)"
            ]
        }
        return workarounds

Key Impossibility Results:

FLP: No async consensus with crash faults
CAP: Can't have all three properties during partitions
Two Generals: Perfect coordination over unreliable links impossible
Byzantine Generals: Need n > 3f for oral messages
Consensus Lower Bounds: f+1 rounds minimum, Ω(n²) messages

⚡ 12. Wait-Free vs Lock-Free Concurrency

Understanding non-blocking synchronization.

import threading
import time
from typing import Optional, Any
import atomic
from dataclasses import dataclass
import sys

class ConcurrencyPrimitive:
    """Base class for concurrency primitives"""

    def __init__(self):
        self.operations = 0
        self.contention = 0

    def benchmark(self, threads: int, operations: int) -> Dict:
        """Benchmark performance under contention"""
        results = {
            'throughput': 0,
            'latency': 0,
            'scalability': 0,
            'fairness': 0
        }
        return results

class LockBasedStack:
    """Stack using locks (blocking)"""

    def __init__(self):
        self.stack = []
        self.lock = threading.Lock()

    def push(self, item: Any):
        with self.lock:
            self.stack.append(item)

    def pop(self) -> Optional[Any]:
        with self.lock:
            if not self.stack:
                return None
            return self.stack.pop()

    def size(self) -> int:
        with self.lock:
            return len(self.stack)

class LockFreeStack:
    """Lock-free stack using CAS (Compare-And-Swap)"""

    @dataclass
    class Node:
        value: Any
        next: Optional['LockFreeStack.Node'] = None

    def __init__(self):
        self.head = None
        # Use atomic reference for head pointer
        self._head = atomic.AtomicReference(self.head)

    def push(self, item: Any):
        new_node = self.Node(item)
        while True:
            current_head = self._head.get()
            new_node.next = current_head

            # Try to swap head pointer
            if self._head.compare_and_set(current_head, new_node):
                return

    def pop(self) -> Optional[Any]:
        while True:
            current_head = self._head.get()
            if current_head is None:
                return None

            next_node = current_head.next

            # Try to swap head pointer
            if self._head.compare_and_set(current_head, next_node):
                return current_head.value

    def size(self) -> int:
        """Non-atomic size (approximate)"""
        count = 0
        current = self._head.get()
        while current:
            count += 1
            current = current.next
        return count

class WaitFreeStack:
    """
    Wait-free stack using helping mechanism

    Wait-free: Every operation completes in bounded number of steps
    regardless of contention
    """

    @dataclass
    class Operation:
        op_type: str  # 'push' or 'pop'
        value: Any = None
        completed: bool = False
        result: Any = None

    def __init__(self):
        self.operations = []  # Shared array for operation descriptors
        self.help_counter = atomic.AtomicInteger(0)

    def push(self, item: Any) -> bool:
        op = self.Operation('push', item)
        self.operations.append(op)

        # Help other operations while waiting
        self._help_others()

        # Complete own operation
        return self._complete_operation(op)

    def pop(self) -> Optional[Any]:
        op = self.Operation('pop')
        self.operations.append(op)

        # Help other operations
        self._help_others()

        # Wait for completion (bounded steps)
        return self._complete_operation(op)

    def _help_others(self):
        """Help other threads complete their operations"""
        for i in range(len(self.operations)):
            op = self.operations[i]
            if not op.completed:
                self._execute_operation(op)

    def _execute_operation(self, op: 'Operation'):
        """Execute an operation"""
        if op.op_type == 'push':
            # Actual push implementation
            # Uses atomic operations
            pass
        elif op.op_type == 'pop':
            # Actual pop implementation
            pass

    def _complete_operation(self, op: 'Operation'):
        """Wait for operation to complete (bounded steps)"""
        steps = 0
        max_steps = 100  # Bound for wait-free guarantee

        while not op.completed and steps < max_steps:
            self._help_others()
            steps += 1
            # Exponential backoff or yield
            time.sleep(0.000001 * (2 ** steps))

        return op.result

class ObstructionFreeStack:
    """
    Obstruction-free stack (weakest non-blocking guarantee)

    Obstruction-free: Guarantees progress only when no contention
    """

    def __init__(self):
        self.stack = []
        self.version = atomic.AtomicInteger(0)

    def push(self, item: Any) -> bool:
        while True:
            # Read current state
            current_version = self.version.get()
            current_stack = self.stack.copy()

            # Modify local copy
            new_stack = current_stack + [item]

            # Try to commit
            if self.version.compare_and_set(current_version, current_version + 1):
                self.stack = new_stack
                return True

            # Contention detected, backoff
            time.sleep(0.000001)

    def pop(self) -> Optional[Any]:
        while True:
            current_version = self.version.get()
            if not self.stack:
                return None

            current_stack = self.stack.copy()
            item = current_stack[-1]
            new_stack = current_stack[:-1]

            if self.version.compare_and_set(current_version, current_version + 1):
                self.stack = new_stack
                return item

            time.sleep(0.000001)

# Performance comparison
class ConcurrencyBenchmark:
    """Benchmark different concurrency approaches"""

    def __init__(self):
        self.results = {}

    def run_benchmark(self, num_threads: int, ops_per_thread: int):
        """Run comprehensive benchmark"""
        print(f"\n=== Concurrency Benchmark ({num_threads} threads, {ops_per_thread} ops each) ===")

        # Test lock-based
        print("\n1. Lock-based stack:")
        lock_time = self._test_stack(LockBasedStack(), num_threads, ops_per_thread)

        # Test lock-free
        print("\n2. Lock-free stack:")
        lockfree_time = self._test_stack(LockFreeStack(), num_threads, ops_per_thread)

        # Test wait-free
        print("\n3. Wait-free stack:")
        waitfree_time = self._test_stack(WaitFreeStack(), num_threads, ops_per_thread)

        # Test obstruction-free
        print("\n4. Obstruction-free stack:")
        obstruction_time = self._test_stack(ObstructionFreeStack(), num_threads, ops_per_thread)

        # Print results
        print(f"\n=== Results ===")
        print(f"Lock-based:     {lock_time:.3f}s")
        print(f"Lock-free:      {lockfree_time:.3f}s")
        print(f"Wait-free:      {waitfree_time:.3f}s")
        print(f"Obstruction-free: {obstruction_time:.3f}s")

        return {
            'lock_based': lock_time,
            'lock_free': lockfree_time,
            'wait_free': waitfree_time,
            'obstruction_free': obstruction_time
        }

    def _test_stack(self, stack, num_threads: int, ops_per_thread: int) -> float:
        """Test specific stack implementation"""
        threads = []
        start_time = time.time()

        def worker(thread_id):
            for i in range(ops_per_thread):
                # Mix of pushes and pops
                if i % 2 == 0:
                    stack.push(f"thread-{thread_id}-{i}")
                else:
                    stack.pop()

        # Create and start threads
        for i in range(num_threads):
            t = threading.Thread(target=worker, args=(i,))
            threads.append(t)
            t.start()

        # Wait for completion
        for t in threads:
            t.join()

        end_time = time.time()
        return end_time - start_time

# Progress guarantee hierarchy
progress_hierarchy = {
    'blocking': {
        'level': 'No progress guarantee',
        'examples': ['mutex', 'semaphore', 'condition variable'],
        'properties': ['Can deadlock', 'Can livelock', 'Priority inversion possible'],
        'use_cases': ['Simple synchronization', 'When fairness needed']
    },
    'obstruction_free': {
        'level': 'Weakest non-blocking',
        'examples': ['Some optimistic algorithms'],
        'properties': ['Progress only with no contention', 'Can starve'],
        'use_cases': ['Low contention scenarios', 'Best-effort systems']
    },
    'lock_free': {
        'level': 'System-wide progress',
        'examples': ['CAS-based structures', 'Atomic operations'],
        'properties': ['Some thread always makes progress', 'Individual threads can starve'],
        'use_cases': ['High contention', 'Real-time systems (soft)']
    },
    'wait_free': {
        'level': 'Strongest non-blocking',
        'examples': ['Helping algorithms', 'Hardware transactions'],
        'properties': ['Every operation completes in bounded steps', 'No starvation'],
        'use_cases': ['Hard real-time', 'Mission critical']
    }
}

# Implementation techniques
class NonBlockingTechniques:
    """Techniques for implementing non-blocking algorithms"""

    @staticmethod
    def compare_and_swap():
        """CAS (Compare-And-Swap) primitive"""
        return {
            'operation': "atomic_compare_exchange_strong(ptr, expected, desired)",
            'effect': "If *ptr == expected, set *ptr = desired, return true",
            'hardware': "Available on all modern CPUs (CMPXCHG on x86)",
            'limitations': "ABA problem, requires careful memory management"
        }

    @staticmethod
    def load_linked_store_conditional():
        """LL/SC (Load-Linked/Store-Conditional)"""
        return {
            'operation': "Two instructions: LL loads, SC stores if no interference",
            'advantage': "No ABA problem",
            'hardware': "ARM, PowerPC, MIPS",
            'limitations': "Can spuriously fail"
        }

    @staticmethod
    def transactional_memory():
        """Hardware Transactional Memory"""
        return {
            'operation': "Atomic block of operations",
            'hardware': "Intel TSX, IBM PowerTM",
            'advantages': ["Simpler programming", "Composability"],
            'limitations': ["Capacity constraints", "Conflict detection overhead"]
        }

    @staticmethod
    def helping_mechanism():
        """Wait-free helping"""
        return {
            'technique': "Threads help each other complete operations",
            'examples': ["Universal constructions", "Herlihy's wait-free consensus"],
            'properties': ["Bounded steps", "High overhead"],
            'use': "When wait-free guarantee is required"
        }

# Example: Non-blocking hash table
class NonBlockingHashTable:
    """Lock-free hash table using CAS"""

    def __init__(self, size: int = 16):
        self.size = size
        self.buckets = [atomic.AtomicReference(None) for _ in range(size)]
        self.resize_lock = atomic.AtomicBoolean(False)

    def _hash(self, key: str) -> int:
        return hash(key) % self.size

    def put(self, key: str, value: Any) -> bool:
        index = self._hash(key)

        while True:
            # Read current bucket head
            current_head = self.buckets[index].get()

            # Check if key already exists
            node = current_head
            while node:
                if node.key == key:
                    # Update existing key
                    new_node = self.Node(key, value, node.next)
                    if self.buckets[index].compare_and_set(current_head, new_node):
                        return True
                    else:
                        # CAS failed, retry
                        break
                node = node.next

            # Key doesn't exist, add new node
            new_node = self.Node(key, value, current_head)
            if self.buckets[index].compare_and_set(current_head, new_node):
                return True

    def get(self, key: str) -> Optional[Any]:
        index = self._hash(key)
        node = self.buckets[index].get()

        while node:
            if node.key == key:
                return node.value
            node = node.next

        return None

    def remove(self, key: str) -> bool:
        index = self._hash(key)

        while True:
            current_head = self.buckets[index].get()

            # Find node to remove
            prev = None
            curr = current_head
            while curr:
                if curr.key == key:
                    # Found it
                    if prev is None:
                        # Removing head
                        new_head = curr.next
                    else:
                        # Removing middle node
                        prev.next = curr.next
                        new_head = current_head

                    if self.buckets[index].compare_and_set(current_head, new_head):
                        return True
                    else:
                        # CAS failed, retry whole operation
                        break

                prev = curr
                curr = curr.next

            # Key not found
            return False

    class Node:
        def __init__(self, key: str, value: Any, next_node: Optional['NonBlockingHashTable.Node'] = None):
            self.key = key
            self.value = value
            self.next = next_node

Concurrency Guarantees Hierarchy:

Blocking: Mutexes, semaphores (can deadlock)
Obstruction-Free: Progress only without contention
Lock-Free: System makes progress (some thread succeeds)
Wait-Free: Every thread completes in bounded steps

📊 13. Queueing Theory: Modeling System Performance

Mathematical models for predicting system behavior under load.

import numpy as np
from scipy import stats, special
from typing import Tuple, List, Dict
import math

class QueueingModels:
    """Mathematical queueing models for performance analysis"""

    @staticmethod
    def mm1_queue(arrival_rate: float, service_rate: float) -> Dict:
        """
        M/M/1 queue: Markovian arrivals, Markovian service, 1 server

        λ = arrival rate (requests/second)
        μ = service rate (requests/second)
        ρ = λ/μ = utilization (must be < 1 for stability)
        """
        if arrival_rate >= service_rate:
            raise ValueError("System unstable: λ >= μ")

        rho = arrival_rate / service_rate

        results = {
            'utilization': rho,
            'expected_customers_system': rho / (1 - rho),
            'expected_customers_queue': rho**2 / (1 - rho),
            'expected_wait_system': 1 / (service_rate - arrival_rate),
            'expected_wait_queue': arrival_rate / (service_rate * (service_rate - arrival_rate)),
            'probability_system_empty': 1 - rho,
            'probability_n_customers': lambda n: (1 - rho) * rho**n
        }

        return results

    @staticmethod
    def mg1_queue(arrival_rate: float, 
                 service_rate: float,
                 service_variance: float) -> Dict:
        """
        M/G/1 queue: Markovian arrivals, General service distribution

        Uses Pollaczek–Khinchine formula
        """
        if arrival_rate >= service_rate:
            raise ValueError("System unstable: λ >= μ")

        rho = arrival_rate / service_rate
        service_time = 1 / service_rate

        # Pollaczek–Khinchine formula
        w_q = (arrival_rate * (service_variance + service_time**2)) / (2 * (1 - rho))
        w = w_q + service_time
        l_q = arrival_rate * w_q
        l = arrival_rate * w

        results = {
            'utilization': rho,
            'expected_customers_queue': l_q,
            'expected_customers_system': l,
            'expected_wait_queue': w_q,
            'expected_wait_system': w,
            'service_time_variance': service_variance,
            'squared_coefficient_variation': service_variance / (service_time**2)
        }

        return results

    @staticmethod
    def gg1_queue(arrival_rate: float,
                 service_rate: float,
                 cv_a: float,  # Coefficient of variation for arrivals
                 cv_s: float) -> Dict:  # Coefficient of variation for service
        """
        G/G/1 queue: General arrivals, General service

        Kingman's approximation formula
        """
        if arrival_rate >= service_rate:
            raise ValueError("System unstable: λ >= μ")

        rho = arrival_rate / service_rate
        service_time = 1 / service_rate

        # Kingman's approximation
        w_q_approx = (rho / (1 - rho)) * ((cv_a**2 + cv_s**2) / 2) * service_time
        w_approx = w_q_approx + service_time
        l_q_approx = arrival_rate * w_q_approx
        l_approx = arrival_rate * w_approx

        results = {
            'utilization': rho,
            'approximate_wait_queue': w_q_approx,
            'approximate_wait_system': w_approx,
            'approximate_customers_queue': l_q_approx,
            'approximate_customers_system': l_approx,
            'arrival_cv': cv_a,
            'service_cv': cv_s,
            'note': 'Results are approximations for G/G/1'
        }

        return results

    @staticmethod
    def mmc_queue(arrival_rate: float, service_rate: float, servers: int) -> Dict:
        """
        M/M/c queue: Multiple servers

        c = number of identical servers
        """
        if arrival_rate >= servers * service_rate:
            raise ValueError(f"System unstable: λ >= {servers}μ")

        rho = arrival_rate / (servers * service_rate)

        # Probability of 0 customers in system
        sum_term = 0
        for n in range(servers):
            sum_term += (arrival_rate / service_rate)**n / math.factorial(n)

        p0 = 1 / (sum_term + 
                  (arrival_rate / service_rate)**servers / 
                  (math.factorial(servers) * (1 - rho)))

        # Probability all servers busy
        c = servers
        p_queue = ((arrival_rate / service_rate)**c / 
                  (math.factorial(c) * (1 - rho))) * p0

        # Expected number in queue
        l_q = (rho * p_queue) / (1 - rho)

        # Expected wait in queue
        w_q = l_q / arrival_rate

        # Expected wait in system
        w = w_q + 1 / service_rate

        # Expected number in system
        l = arrival_rate * w

        results = {
            'utilization_per_server': rho,
            'total_utilization': arrival_rate / service_rate,
            'probability_all_servers_busy': p_queue,
            'expected_customers_queue': l_q,
            'expected_customers_system': l,
            'expected_wait_queue': w_q,
            'expected_wait_system': w,
            'probability_system_empty': p0
        }

        return results

class NetworkOfQueues:
    """Models for networks of queues (Jackson networks)"""

    @staticmethod
    def jackson_network(service_rates: List[float],
                       routing_matrix: np.ndarray,
                       arrival_rates: List[float]) -> Dict:
        """
        Jackson network: Open network of M/M/1 queues

        routing_matrix[i][j] = probability going from i to j
        routing_matrix[i][0] = probability leaving system from i
        """
        n = len(service_rates)

        # Solve traffic equations: λ_i = γ_i + Σ_j λ_j * p_ji
        # Where γ_i is external arrival rate to node i

        # Set up equations: λ - λP = γ
        P = routing_matrix
        I = np.eye(n)

        # Solve linear equations
        A = I - P.T  # Transpose because equations are different
        lambda_vec = np.linalg.solve(A, arrival_rates)

        # Check stability: λ_i < μ_i for all i
        stable = all(lambda_i < mu_i for lambda_i, mu_i in zip(lambda_vec, service_rates))

        if not stable:
            raise ValueError("Network unstable: Some nodes have λ >= μ")

        # Calculate performance for each node
        node_performance = []
        for i in range(n):
            lambda_i = lambda_vec[i]
            mu_i = service_rates[i]
            rho_i = lambda_i / mu_i

            node_result = {
                'arrival_rate': lambda_i,
                'service_rate': mu_i,
                'utilization': rho_i,
                'expected_customers': rho_i / (1 - rho_i),
                'expected_wait': 1 / (mu_i - lambda_i)
            }
            node_performance.append(node_result)

        # Overall network performance
        total_arrival = sum(arrival_rates)
        total_departure = sum(lambda_vec * (1 - np.sum(P, axis=1)))  # Leaving system

        results = {
            'stable': stable,
            'node_performance': node_performance,
            'total_throughput': total_departure,
            'total_arrival': total_arrival,
            'utilization_vector': [p['utilization'] for p in node_performance],
            'bottleneck_node': np.argmax([p['utilization'] for p in node_performance])
        }

        return results

    @staticmethod
    def mean_value_analysis(service_demands: List[float],
                           think_time: float,
                           users: int) -> Dict:
        """
        Mean Value Analysis for closed queueing networks

        Used for modeling systems with fixed number of users
        (e.g., database connection pools)
        """
        n = len(service_demands)  # Number of devices (queues)
        N = users  # Number of users

        # Initialize
        response_times = [0] * n
        throughputs = [0] * n
        queue_lengths = [0] * n

        # Iterative solution
        for k in range(1, N + 1):
            # Calculate response times
            for i in range(n):
                response_times[i] = service_demands[i] * (1 + queue_lengths[i])

            # Calculate system response time
            R = sum(response_times)

            # Calculate throughput
            X = k / (think_time + R)

            # Calculate queue lengths
            for i in range(n):
                queue_lengths[i] = X * response_times[i]

        results = {
            'users': N,
            'throughput': X,
            'system_response_time': R,
            'device_response_times': response_times,
            'device_queue_lengths': queue_lengths,
            'device_utilizations': [X * d for d in service_demands]
        }

        return results

class PerformanceMetrics:
    """Key performance metrics from queueing theory"""

    @staticmethod
    def little_law(arrival_rate: float, 
                  response_time: float,
                  queue_length: float = None) -> Dict:
        """
        Little's Law: L = λW

        L = average number in system
        λ = arrival rate
        W = average time in system
        """
        if queue_length is None:
            # Calculate L from λ and W
            L = arrival_rate * response_time
            return {'customers_in_system': L}
        elif response_time is None:
            # Calculate W from λ and L
            W = queue_length / arrival_rate
            return {'response_time': W}
        else:
            # Calculate λ from L and W
            lambda_calc = queue_length / response_time
            return {'arrival_rate': lambda_calc}

    @staticmethod
    def utilization_law(arrival_rate: float,
                       service_rate: float,
                       servers: int = 1) -> float:
        """
        Utilization Law: ρ = λ / (cμ)

        ρ = utilization
        c = number of servers
        """
        return arrival_rate / (servers * service_rate)

    @staticmethod
    def response_time_law(service_time: float,
                         utilization: float,
                         cv_a: float = 1.0,
                         cv_s: float = 1.0) -> float:
        """
        Response time approximation

        R = S / (1 - ρ) for M/M/1
        More generally: R ≈ S * (1 + (ρ/(1-ρ)) * ((cv_a² + cv_s²)/2))
        """
        if utilization >= 1:
            return float('inf')

        # Kingman's approximation
        return service_time * (1 + (utilization / (1 - utilization)) * 
                              ((cv_a**2 + cv_s**2) / 2))

    @staticmethod
    def throughput_bounds(service_rate: float,
                         servers: int,
                         think_time: float,
                         users: int) -> Tuple[float, float]:
        """
        Asymptotic bounds on throughput

        Lower bound: X ≥ N / (Z + ΣD_i)
        Upper bound: X ≤ min(1/D_max, N/(Z + D_max))

        Where D_i = service demand at device i
        Z = think time
        N = number of users
        """
        # For single device case
        D_max = 1 / service_rate  # Maximum service demand

        lower_bound = users / (think_time + D_max)
        upper_bound = min(service_rate * servers, 
                         users / (think_time + D_max))

        return lower_bound, upper_bound

# Example: Modeling a web service
class WebServiceModel:
    """Queueing model for a web service"""

    def __init__(self, config: Dict):
        self.config = config

        # Service components
        self.frontend_rate = config.get('frontend_rate', 1000)  # req/s
        self.backend_rate = config.get('backend_rate', 500)     # req/s
        self.database_rate = config.get('database_rate', 200)   # req/s

        # Network delays
        self.network_delay = config.get('network_delay', 0.010)  # 10ms

        # Routing probabilities
        self.routing = config.get('routing', {
            'frontend_to_backend': 0.8,
            'frontend_to_database': 0.2,
            'backend_to_database': 0.6,
            'backend_to_frontend': 0.4
        })

    def model_performance(self, arrival_rate: float) -> Dict:
        """Model system performance under given load"""

        # Frontend queue (M/M/1)
        frontend = QueueingModels.mm1_queue(
            arrival_rate=arrival_rate,
            service_rate=self.frontend_rate
        )

        # Backend queue (M/G/1 with variable service times)
        backend_arrival = arrival_rate * self.routing['frontend_to_backend']
        backend = QueueingModels.mg1_queue(
            arrival_rate=backend_arrival,
            service_rate=self.backend_rate,
            service_variance=0.001  # Example variance
        )

        # Database queue (M/M/c with connection pooling)
        database_arrival = (arrival_rate * self.routing['frontend_to_database'] +
                           backend_arrival * self.routing['backend_to_database'])
        database = QueueingModels.mmc_queue(
            arrival_rate=database_arrival,
            service_rate=self.database_rate,
            servers=50  # Connection pool size
        )

        # Total response time
        total_response = (
            frontend['expected_wait_system'] +
            backend['expected_wait_system'] +
            database['expected_wait_system'] +
            self.network_delay * 3  # Three network hops
        )

        # System throughput (limited by bottleneck)
        bottleneck_utilization = max(
            frontend['utilization'],
            backend['utilization'],
            database['utilization_per_server']
        )

        # Capacity planning
        max_capacity = min(
            self.frontend_rate * 0.8,  # 80% utilization target
            self.backend_rate * 0.8,
            self.database_rate * 50 * 0.8  # 50 connections * 80%
        )

        results = {
            'arrival_rate': arrival_rate,
            'total_response_time': total_response,
            'frontend_response': frontend['expected_wait_system'],
            'backend_response': backend['expected_wait_system'],
            'database_response': database['expected_wait_system'],
            'bottleneck_utilization': bottleneck_utilization,
            'max_recommended_capacity': max_capacity,
            'predicted_p95_response': total_response * 2.0,  # Approximation
            'predicted_p99_response': total_response * 3.0   # Approximation
        }

        return results

    def sensitivity_analysis(self, 
                            arrival_rates: List[float]) -> List[Dict]:
        """Analyze performance across different load levels"""
        results = []

        for rate in arrival_rates:
            try:
                perf = self.model_performance(rate)
                perf['stable'] = True
            except ValueError as e:
                # System unstable at this rate
                perf = {
                    'arrival_rate': rate,
                    'stable': False,
                    'error': str(e)
                }

            results.append(perf)

        return results

    def find_saturation_point(self, 
                             target_response: float = 1.0,
                             max_rate: float = 10000) -> float:
        """Find arrival rate where response time exceeds target"""
        low = 0
        high = max_rate

        for _ in range(20):  # Binary search
            mid = (low + high) / 2
            try:
                perf = self.model_performance(mid)
                if perf['total_response_time'] > target_response:
                    high = mid
                else:
                    low = mid
            except ValueError:
                # Unstable at mid rate
                high = mid

        return (low + high) / 2

# Practical applications
class QueueingApplications:
    """Real applications of queueing theory"""

    @staticmethod
    def capacity_planning(current_load: float,
                         growth_rate: float,
                         sla_response: float,
                         model: WebServiceModel) -> Dict:
        """Plan capacity based on growth projections"""

        # Project load over time
        months = 12
        monthly_loads = [current_load * (1 + growth_rate)**i 
                        for i in range(months + 1)]

        # Find when SLA will be violated
        violation_month = None
        for month, load in enumerate(monthly_loads):
            try:
                perf = model.model_performance(load)
                if perf['total_response_time'] > sla_response:
                    violation_month = month
                    break
            except ValueError:
                violation_month = month
                break

        # Recommendation
        if violation_month is None:
            recommendation = f"Current capacity sufficient for {months} months"
        else:
            recommendation = (f"Need capacity upgrade before month "
                            f"{violation_month} (load: {monthly_loads[violation_month]:.0f} req/s)")

        return {
            'current_load': current_load,
            'growth_rate': growth_rate,
            'sla_response': sla_response,
            'violation_month': violation_month,
            'recommendation': recommendation,
            'projected_loads': monthly_loads
        }

    @staticmethod
    def autoscaling_policy(utilization_target: float = 0.7,
                          scale_out_threshold: float = 0.8,
                          scale_in_threshold: float = 0.4,
                          cooldown_period: int = 300) -> Dict:
        """Design autoscaling policy based on queueing theory"""

        policy = {
            'metrics': [
                'CPU utilization (smoothed over 5 minutes)',
                'Request queue length',
                'Average response time',
                'Error rate'
            ],
            'scale_out_conditions': [
                f'Utilization > {scale_out_threshold} for 3 consecutive minutes',
                f'Queue length > 100 for 2 minutes',
                f'P95 response time > SLA for 5 minutes'
            ],
            'scale_in_conditions': [
                f'Utilization < {scale_in_threshold} for 10 minutes',
                f'Queue length < 10 for 5 minutes'
            ],
            'cooldown_period': cooldown_period,
            'max_instances': 'Based on cost budget',
            'min_instances': 'Based on baseline load'
        }

        # Calculate optimal thresholds using queueing theory
        # For M/M/c, want ρ ≈ utilization_target
        optimal_servers = lambda arrival_rate, service_rate: math.ceil(
            arrival_rate / (service_rate * utilization_target)
        )

        policy['optimal_scaling'] = optimal_servers
        return policy

    @staticmethod
    def sla_compliance(observed_response_times: List[float],
                      sla_p95: float,
                      sla_p99: float) -> Dict:
        """Check SLA compliance using queueing theory predictions"""

        # Empirical percentiles
        sorted_times = np.sort(observed_response_times)
        n = len(sorted_times)

        p95_empirical = sorted_times[int(0.95 * n)]
        p99_empirical = sorted_times[int(0.99 * n)]

        # Fit distribution (exponential for M/M/1, normal for heavy tail)
        mean_response = np.mean(observed_response_times)

        # For exponential distribution: p95 = mean * -ln(0.05) ≈ mean * 3
        # p99 = mean * -ln(0.01) ≈ mean * 4.6
        p95_predicted = mean_response * 3
        p99_predicted = mean_response * 4.6

        compliance = {
            'observed_p95': p95_empirical,
            'observed_p99': p99_empirical,
            'predicted_p95': p95_predicted,
            'predicted_p99': p99_predicted,
            'sla_p95_violation': p95_empirical > sla_p95,
            'sla_p99_violation': p99_empirical > sla_p99,
            'distribution_fit': 'Exponential (M/M/1 approximation)',
            'recommendation': 'Consider G/G/1 model if heavy tail observed'
        }

        return compliance

Queueing Theory Key Formulas:

Little's Law: L = λW
Utilization Law: ρ = λ/μ
M/M/1 Formulas:
- L = ρ/(1-ρ)
- W = 1/(μ-λ)
Pollaczek–Khinchine (M/G/1):
- W_q = λ(σ² + 1/μ²)/(2(1-ρ))
Kingman's Approximation (G/G/1):
- W_q ≈ (ρ/(1-ρ)) * ((c_a² + c_s²)/2) * (1/μ)

🎲 14. Power of d Choices: Intelligent Load Balancing

Theoretical foundation for smarter load balancing algorithms.

import random
import numpy as np
from typing import List, Dict, Tuple
import math
from collections import Counter, deque

class PowerOfDChoices:
    """
    The Power of d Choices load balancing

    Instead of random assignment (d=1) or checking all (d=n),
    check d random servers and pick least loaded
    """

    def __init__(self, n_servers: int, d: int = 2):
        self.n_servers = n_servers
        self.d = d
        self.loads = [0] * n_servers
        self.queue_lengths = [0] * n_servers

        # Statistics
        self.requests_processed = 0
        self.load_history = []

    def dispatch_request(self, request_size: int = 1) -> int:
        """Dispatch request using power of d choices"""
        # Sample d servers randomly
        sampled_indices = random.sample(range(self.n_servers), self.d)

        # Find least loaded among sampled
        min_load = float('inf')
        selected_server = sampled_indices[0]

        for idx in sampled_indices:
            if self.loads[idx] < min_load:
                min_load = self.loads[idx]
                selected_server = idx

        # Assign request to selected server
        self.loads[selected_server] += request_size
        self.queue_lengths[selected_server] += 1

        self.requests_processed += 1

        # Record load distribution periodically
        if self.requests_processed % 100 == 0:
            self.load_history.append(self.loads.copy())

        return selected_server

    def process_request(self, server_idx: int, service_time: int = 1):
        """Simulate request processing"""
        if self.loads[server_idx] > 0:
            self.loads[server_idx] -= min(service_time, self.loads[server_idx])
            self.queue_lengths[server_idx] = max(0, self.queue_lengths[server_idx] - 1)

    def get_load_statistics(self) -> Dict:
        """Get statistics about load distribution"""
        loads = self.loads
        queue_lengths = self.queue_lengths

        stats = {
            'mean_load': np.mean(loads),
            'std_load': np.std(loads),
            'max_load': max(loads),
            'min_load': min(loads),
            'load_imbalance': max(loads) - min(loads),
            'mean_queue_length': np.mean(queue_lengths),
            'max_queue_length': max(queue_lengths),
            'empty_servers': sum(1 for l in loads if l == 0),
            'high_load_servers': sum(1 for l in loads if l > 2 * np.mean(loads))
        }

        # Gini coefficient for load inequality
        sorted_loads = np.sort(loads)
        n = len(sorted_loads)
        cumulative = np.cumsum(sorted_loads)
        gini = (n + 1 - 2 * np.sum(cumulative) / cumulative[-1]) / n
        stats['gini_coefficient'] = gini

        return stats

    def theoretical_improvement(self) -> float:
        """
        Theoretical improvement over random assignment

        For d=2, maximum load is O(log log n / log 2) vs O(log n) for d=1
        """
        if self.d == 1:
            # Random assignment
            return np.log(self.n_servers) / np.log(np.log(self.n_servers))
        else:
            # Power of d choices
            return np.log(np.log(self.n_servers)) / np.log(self.d)

class AdaptivePowerOfD:
    """
    Adaptive d based on system load

    Idea: Increase d when system is heavily loaded,
    decrease d when lightly loaded to reduce sampling overhead
    """

    def __init__(self, n_servers: int, max_d: int = 5):
        self.n_servers = n_servers
        self.max_d = max_d
        self.loads = [0] * n_servers

        # Adaptive parameters
        self.current_d = 2
        self.sampling_history = deque(maxlen=100)

    def adaptive_dispatch(self, request_size: int = 1) -> int:
        """Dispatch with adaptive d"""
        # Update d based on system state
        self._update_d()

        # Use current d for sampling
        if self.current_d >= self.n_servers:
            # Check all servers if d >= n
            sampled_indices = list(range(self.n_servers))
        else:
            sampled_indices = random.sample(range(self.n_servers), self.current_d)

        # Find least loaded
        min_load = float('inf')
        selected = sampled_indices[0]

        for idx in sampled_indices:
            if self.loads[idx] < min_load:
                min_load = self.loads[idx]
                selected = idx

        # Assign request
        self.loads[selected] += request_size
        self.sampling_history.append(self.current_d)

        return selected

    def _update_d(self):
        """Adapt d based on system load"""
        mean_load = np.mean(self.loads)
        load_std = np.std(self.loads)

        # High load or high variance → increase d
        if mean_load > 0.7 * self.n_servers or load_std > 0.3 * mean_load:
            self.current_d = min(self.current_d + 1, self.max_d)
        # Low load and low variance → decrease d
        elif mean_load < 0.3 * self.n_servers and load_std < 0.1 * mean_load:
            self.current_d = max(self.current_d - 1, 1)

    def get_adaptation_stats(self) -> Dict:
        """Get statistics about d adaptation"""
        if not self.sampling_history:
            return {'current_d': self.current_d}

        history = list(self.sampling_history)
        return {
            'current_d': self.current_d,
            'mean_d': np.mean(history),
            'std_d': np.std(history),
            'min_d': min(history),
            'max_d': max(history),
            'adaptation_frequency': len(set(history)) / len(history)
        }

class WeightedPowerOfD:
    """
    Weighted power of d choices

    Servers have different capacities/weights
    Choose server with minimum load/weight ratio
    """

    def __init__(self, server_weights: List[float]):
        self.weights = server_weights
        self.n_servers = len(server_weights)
        self.loads = [0] * self.n_servers

    def weighted_dispatch(self, request_size: float, d: int = 2) -> int:
        """Dispatch considering server weights"""
        sampled = random.sample(range(self.n_servers), min(d, self.n_servers))

        # Find server with minimum load/weight ratio
        min_ratio = float('inf')
        selected = sampled[0]

        for idx in sampled:
            ratio = self.loads[idx] / self.weights[idx]
            if ratio < min_ratio:
                min_ratio = ratio
                selected = idx

        self.loads[selected] += request_size
        return selected

    def get_weighted_stats(self) -> Dict:
        """Statistics considering weights"""
        ratios = [l / w for l, w in zip(self.loads, self.weights)]

        return {
            'mean_ratio': np.mean(ratios),
            'std_ratio': np.std(ratios),
            'max_ratio': max(ratios),
            'min_ratio': min(ratios),
            'fairness_index': self._jain_fairness_index(ratios)
        }

    def _jain_fairness_index(self, values: List[float]) -> float:
        """Jain's fairness index"""
        n = len(values)
        numerator = sum(values) ** 2
        denominator = n * sum(v**2 for v in values)
        return numerator / denominator if denominator > 0 else 0

class ConsistentHashingWithLoad:
    """
    Combine consistent hashing with power of d choices

    First use consistent hashing to map request to servers,
    then use power of d choices among neighboring servers
    """

    def __init__(self, n_servers: int, virtual_nodes: int = 100, d: int = 2):
        self.n_servers = n_servers
        self.virtual_nodes = virtual_nodes
        self.d = d

        # Consistent hashing ring
        self.ring = {}
        self.sorted_keys = []

        # Server loads
        self.loads = [0] * n_servers

        # Initialize ring
        self._initialize_ring()

    def _initialize_ring(self):
        """Initialize consistent hashing ring"""
        for server in range(self.n_servers):
            for vnode in range(self.virtual_nodes):
                key = hash(f"{server}-{vnode}") % (2**32)
                self.ring[key] = server
                self.sorted_keys.append(key)

        self.sorted_keys.sort()

    def dispatch(self, request_id: str, request_size: int = 1) -> int:
        """Dispatch using consistent hashing + power of d"""
        # Hash request
        request_hash = hash(request_id) % (2**32)

        # Find primary server (consistent hashing)
        primary_idx = self._find_server(request_hash)

        # Get d-1 neighboring servers
        neighbors = self._get_neighbors(primary_idx, self.d - 1)

        # Consider primary + neighbors
        candidates = [primary_idx] + neighbors

        # Choose least loaded among candidates
        min_load = float('inf')
        selected = primary_idx

        for server in candidates:
            if self.loads[server] < min_load:
                min_load = self.loads[server]
                selected = server

        # Assign request
        self.loads[selected] += request_size
        return selected

    def _find_server(self, key: int) -> int:
        """Find server for key using consistent hashing"""
        if not self.sorted_keys:
            return 0

        # Binary search for key
        import bisect
        idx = bisect.bisect_left(self.sorted_keys, key)

        if idx == len(self.sorted_keys):
            idx = 0

        return self.ring[self.sorted_keys[idx]]

    def _get_neighbors(self, server_idx: int, count: int) -> List[int]:
        """Get neighboring servers in the ring"""
        # Find position of server's virtual nodes
        server_vnodes = [k for k, v in self.ring.items() if v == server_idx]

        if not server_vnodes:
            return []

        # Take first virtual node as reference
        ref_vnode = server_vnodes[0]
        ref_idx = self.sorted_keys.index(ref_vnode)

        # Get neighbors clockwise
        neighbors = []
        for offset in range(1, len(self.sorted_keys)):
            neighbor_idx = (ref_idx + offset) % len(self.sorted_keys)
            neighbor_server = self.ring[self.sorted_keys[neighbor_idx]]

            if neighbor_server != server_idx and neighbor_server not in neighbors:
                neighbors.append(neighbor_server)
                if len(neighbors) >= count:
                    break

        return neighbors

    def add_server(self):
        """Add new server to the ring"""
        new_server = self.n_servers
        self.n_servers += 1
        self.loads.append(0)

        # Add virtual nodes for new server
        for vnode in range(self.virtual_nodes):
            key = hash(f"{new_server}-{vnode}") % (2**32)
            self.ring[key] = new_server
            self.sorted_keys.append(key)

        self.sorted_keys.sort()

        # Rebalance some load (in practice)
        return new_server

    def remove_server(self, server_idx: int):
        """Remove server from the ring"""
        # Remove virtual nodes
        keys_to_remove = [k for k, v in self.ring.items() if v == server_idx]
        for key in keys_to_remove:
            del self.ring[key]
            self.sorted_keys.remove(key)

        # Redistribute load (simplified)
        total_load = self.loads[server_idx]
        avg_redistribution = total_load / (self.n_servers - 1)

        for i in range(self.n_servers):
            if i != server_idx:
                self.loads[i] += avg_redistribution

        # Remove server
        self.loads.pop(server_idx)
        self.n_servers -= 1

        # Update server indices in ring
        new_ring = {}
        for key, server in self.ring.items():
            if server > server_idx:
                new_ring[key] = server - 1
            else:
                new_ring[key] = server

        self.ring = new_ring

# Theoretical analysis
class PowerOfDAnalysis:
    """Theoretical analysis of power of d choices"""

    @staticmethod
    def expected_max_load(n: int, d: int, m: int) -> float:
        """
        Expected maximum load with power of d choices

        n = number of servers (bins)
        m = number of requests (balls)
        d = number of choices

        For d=1 (random): O(log n / log log n)
        For d=2: O(log log n / log 2)
        For d≥2: O(log log n / log d)
        """
        if d == 1:
            # Classical balls and bins
            return np.log(n) / np.log(np.log(n)) * (m / n)
        else:
            return np.log(np.log(n)) / np.log(d) * (m / n)

    @staticmethod
    def load_imbalance_reduction(d: int) -> float:
        """
        How much load imbalance reduces with d

        Returns: Factor reduction in maximum load compared to random
        """
        # Approximate reduction factor
        return np.log(d) / np.log(2)

    @staticmethod
    def optimal_d(n: int, sampling_cost: float, imbalance_cost: float) -> int:
        """
        Optimal d balancing sampling cost vs load imbalance

        sampling_cost: Cost to sample one server
        imbalance_cost: Cost per unit of load imbalance
        """
        # Trade-off: More d reduces imbalance but increases sampling
        optimal = 1

        for d in range(2, n + 1):
            # Expected imbalance with d choices
            imbalance = PowerOfDAnalysis.expected_max_load(n, d, n)
            imbalance_d1 = PowerOfDAnalysis.expected_max_load(n, 1, n)
            imbalance_reduction = imbalance_d1 - imbalance

            # Cost benefit
            sampling_overhead = d * sampling_cost
            imbalance_benefit = imbalance_reduction * imbalance_cost

            if imbalance_benefit > sampling_overhead:
                optimal = d
            else:
                break

        return optimal

# Simulation and comparison
class LoadBalancingSimulator:
    """Simulate different load balancing strategies"""

    def __init__(self, n_servers: int, total_requests: int):
        self.n_servers = n_servers
        self.total_requests = total_requests

    def simulate_strategy(self, strategy: str, **kwargs) -> Dict:
        """Simulate a specific load balancing strategy"""
        if strategy == 'random':
            return self._simulate_random()
        elif strategy == 'round_robin':
            return self._simulate_round_robin()
        elif strategy == 'least_loaded':
            return self._simulate_least_loaded()
        elif strategy == 'power_of_d':
            d = kwargs.get('d', 2)
            return self._simulate_power_of_d(d)
        elif strategy == 'consistent_hashing':
            return self._simulate_consistent_hashing()
        elif strategy == 'weighted':
            weights = kwargs.get('weights', [1] * self.n_servers)
            return self._simulate_weighted(weights)
        else:
            raise ValueError(f"Unknown strategy: {strategy}")

    def _simulate_random(self) -> Dict:
        """Random assignment"""
        loads = [0] * self.n_servers

        for _ in range(self.total_requests):
            server = random.randint(0, self.n_servers - 1)
            loads[server] += 1

        return self._analyze_loads(loads, 'random')

    def _simulate_round_robin(self) -> Dict:
        """Round robin assignment"""
        loads = [0] * self.n_servers
        current = 0

        for _ in range(self.total_requests):
            loads[current] += 1
            current = (current + 1) % self.n_servers

        return self._analyze_loads(loads, 'round_robin')

    def _simulate_least_loaded(self) -> Dict:
        """Always choose least loaded server (check all)"""
        loads = [0] * self.n_servers

        for _ in range(self.total_requests):
            # Find least loaded server
            min_load = min(loads)
            min_servers = [i for i, l in enumerate(loads) if l == min_load]

            # Choose randomly among least loaded
            server = random.choice(min_servers)
            loads[server] += 1

        return self._analyze_loads(loads, 'least_loaded')

    def _simulate_power_of_d(self, d: int) -> Dict:
        """Power of d choices"""
        loads = [0] * self.n_servers

        for _ in range(self.total_requests):
            # Sample d servers
            sampled = random.sample(range(self.n_servers), min(d, self.n_servers))

            # Find least loaded among sampled
            min_load = min(loads[i] for i in sampled)
            min_sampled = [i for i in sampled if loads[i] == min_load]

            server = random.choice(min_sampled)
            loads[server] += 1

        return self._analyze_loads(loads, f'power_of_{d}')

    def _simulate_consistent_hashing(self) -> Dict:
        """Consistent hashing"""
        loads = [0] * self.n_servers

        # Simple consistent hashing simulation
        ring_size = 1000
        ring = [i % self.n_servers for i in range(ring_size)]

        for i in range(self.total_requests):
            request_hash = hash(str(i)) % ring_size
            server = ring[request_hash]
            loads[server] += 1

        return self._analyze_loads(loads, 'consistent_hashing')

    def _simulate_weighted(self, weights: List[float]) -> Dict:
        """Weighted round robin"""
        loads = [0] * self.n_servers

        # Normalize weights
        total_weight = sum(weights)
        normalized = [w / total_weight for w in weights]

        # Cumulative distribution
        cumulative = []
        current = 0
        for w in normalized:
            current += w
            cumulative.append(current)

        for i in range(self.total_requests):
            r = random.random()
            for server, cum in enumerate(cumulative):
                if r <= cum:
                    loads[server] += 1
                    break

        return self._analyze_loads(loads, 'weighted')

    def _analyze_loads(self, loads: List[int], strategy: str) -> Dict:
        """Analyze load distribution"""
        mean_load = np.mean(loads)
        std_load = np.std(loads)
        max_load = max(loads)
        min_load = min(loads)

        # Imbalance metrics
        imbalance_ratio = max_load / mean_load if mean_load > 0 else float('inf')
        cv = std_load / mean_load if mean_load > 0 else 0  # Coefficient of variation

        return {
            'strategy': strategy,
            'mean_load': mean_load,
            'std_load': std_load,
            'max_load': max_load,
            'min_load': min_load,
            'imbalance_ratio': imbalance_ratio,
            'coefficient_of_variation': cv,
            'empty_servers': sum(1 for l in loads if l == 0),
            'loads': loads.copy()
        }

    def compare_strategies(self, strategies: List[Tuple[str, Dict]]) -> List[Dict]:
        """Compare multiple strategies"""
        results = []

        for strategy_name, kwargs in strategies:
            result = self.simulate_strategy(strategy_name, **kwargs)
            results.append(result)

        # Sort by imbalance ratio (lower is better)
        results.sort(key=lambda x: x['imbalance_ratio'])

        return results

Power of d Choices Insights:

d=1: Random assignment → O(log n) maximum load
d=2: Check two, pick least loaded → O(log log n) maximum load
d=log n: Check logarithmic number → Constant factor imbalance
d=n: Check all (least loaded) → Perfect balance

Key Theoretical Result: Checking just d=2 provides exponential improvement over random assignment, with most benefit achieved by d=2-5.

🔗 15. Consistent Hashing Theory

Mathematical analysis of consistent hashing properties.

import hashlib
import bisect
from typing import List, Dict, Set, Tuple
import numpy as np
import math
from collections import defaultdict

class ConsistentHashingTheory:
    """Theoretical analysis of consistent hashing"""

    @staticmethod
    def load_distribution(n_nodes: int, 
                         n_items: int,
                         vnodes_per_node: int = 100) -> Dict:
        """
        Analyze load distribution in consistent hashing

        Returns: Statistical properties of load distribution
        """
        # Simulate consistent hashing
        ring_size = 2**32
        ring = []
        node_positions = defaultdict(list)

        # Place virtual nodes
        for node in range(n_nodes):
            for vnode in range(vnodes_per_node):
                position = ConsistentHashingTheory._hash_position(
                    f"node{node}-vnode{vnode}", ring_size
                )
                ring.append(position)
                node_positions[node].append(position)

        ring.sort()

        # Simulate item placement
        loads = [0] * n_nodes

        for item in range(n_items):
            position = ConsistentHashingTheory._hash_position(
                f"item{item}", ring_size
            )

            # Find node responsible
            idx = bisect.bisect_left(ring, position)
            if idx == len(ring):
                idx = 0

            # Determine which node owns this position
            for node, positions in node_positions.items():
                if ring[idx] in positions:
                    loads[node] += 1
                    break

        # Analyze distribution
        mean_load = n_items / n_nodes
        std_load = np.std(loads)
        cv = std_load / mean_load if mean_load > 0 else 0

        # Theoretical bounds
        # With k virtual nodes per server, load per server is O((log n)/k)
        theoretical_std = math.sqrt(mean_load / vnodes_per_node)

        return {
            'actual_mean': mean_load,
            'actual_std': std_load,
            'actual_cv': cv,
            'theoretical_std': theoretical_std,
            'max_load': max(loads),
            'min_load': min(loads),
            'imbalance_ratio': max(loads) / mean_load,
            'loads': loads
        }

    @staticmethod
    def movement_cost(n_old: int, 
                     n_new: int,
                     vnodes_per_node: int = 100,
                     total_items: int = 100000) -> float:
        """
        Calculate fraction of items that move when adding/removing nodes

        In consistent hashing, only O(1/n) items need to move
        """
        ring_size = 2**32

        # Create old ring
        old_ring = []
        old_nodes = {}

        for node in range(n_old):
            for vnode in range(vnodes_per_node):
                pos = ConsistentHashingTheory._hash_position(
                    f"old{node}-{vnode}", ring_size
                )
                old_ring.append(pos)
                old_nodes[pos] = node

        old_ring.sort()

        # Create new ring (add one node)
        new_ring = []
        new_nodes = {}

        for node in range(n_new):
            for vnode in range(vnodes_per_node):
                pos = ConsistentHashingTheory._hash_position(
                    f"new{node}-{vnode}", ring_size
                )
                new_ring.append(pos)
                new_nodes[pos] = node

        new_ring.sort()

        # Calculate movement
        moved_items = 0

        for item in range(total_items):
            item_pos = ConsistentHashingTheory._hash_position(
                f"item{item}", ring_size
            )

            # Find responsible node in old ring
            old_idx = bisect.bisect_left(old_ring, item_pos)
            if old_idx == len(old_ring):
                old_idx = 0
            old_node = old_nodes[old_ring[old_idx]]

            # Find responsible node in new ring
            new_idx = bisect.bisect_left(new_ring, item_pos)
            if new_idx == len(new_ring):
                new_idx = 0
            new_node = new_nodes[new_ring[new_idx]]

            if old_node != new_node:
                moved_items += 1

        fraction_moved = moved_items / total_items
        theoretical_moved = 1 / (n_old + 1)  # When adding one node

        return {
            'actual_fraction_moved': fraction_moved,
            'theoretical_fraction': theoretical_moved,
            'absolute_moved': moved_items,
            'efficiency': theoretical_moved / fraction_moved if fraction_moved > 0 else 0
        }

    @staticmethod
    def virtual_nodes_analysis(n_nodes: int,
                              n_items: int,
                              vnodes_range: List[int]) -> List[Dict]:
        """
        Analyze effect of virtual nodes on load balance
        """
        results = []

        for vnodes in vnodes_range:
            dist = ConsistentHashingTheory.load_distribution(
                n_nodes, n_items, vnodes
            )

            results.append({
                'virtual_nodes': vnodes,
                'coefficient_of_variation': dist['actual_cv'],
                'imbalance_ratio': dist['imbalance_ratio'],
                'theoretical_std': dist['theoretical_std'],
                'actual_std': dist['actual_std']
            })

        return results

    @staticmethod
    def bounded_loads(n_nodes: int,
                     capacity: float,
                     vnodes_per_node: int = 100) -> Dict:
        """
        Analyze bounded-load consistent hashing

        Each node has capacity C, algorithm skips full nodes
        """
        ring_size = 2**32
        ring = []
        node_loads = [0] * n_nodes
        node_capacities = [capacity] * n_nodes

        # Initialize ring
        for node in range(n_nodes):
            for vnode in range(vnodes_per_node):
                pos = ConsistentHashingTheory._hash_position(
                    f"node{node}-{vnode}", ring_size
                )
                ring.append((pos, node))

        ring.sort(key=lambda x: x[0])

        # Simulate with bounded loads
        rejected = 0
        total_items = 10000

        for item in range(total_items):
            pos = ConsistentHashingTheory._hash_position(
                f"item{item}", ring_size
            )

            # Find position in ring
            idx = bisect.bisect_left([r[0] for r in ring], pos)
            if idx == len(ring):
                idx = 0

            # Try nodes in order until finding one with capacity
            placed = False
            attempts = 0
            max_attempts = len(ring)  # Try entire ring

            while not placed and attempts < max_attempts:
                current_idx = (idx + attempts) % len(ring)
                node = ring[current_idx][1]

                if node_loads[node] < node_capacities[node]:
                    node_loads[node] += 1
                    placed = True
                    break

                attempts += 1

            if not placed:
                rejected += 1

        utilization = sum(node_loads) / (n_nodes * capacity)
        rejection_rate = rejected / total_items

        return {
            'utilization': utilization,
            'rejection_rate': rejection_rate,
            'node_loads': node_loads,
            'max_load': max(node_loads),
            'min_load': min(node_loads),
            'load_std': np.std(node_loads)
        }

    @staticmethod
    def _hash_position(key: str, ring_size: int) -> int:
        """Hash key to position on ring"""
        # Using MD5 for good distribution
        hash_obj = hashlib.md5(key.encode())
        hash_int = int(hash_obj.hexdigest(), 16)
        return hash_int % ring_size

class RendezvousHashing:
    """
    Rendezvous (Highest Random Weight) hashing

    Alternative to consistent hashing with better load distribution
    but O(n) computation for each key
    """

    def __init__(self, nodes: List[str]):
        self.nodes = nodes

    def assign(self, key: str) -> str:
        """
        Assign key to node with highest weight

        weight = hash(key + node)
        Choose node with maximum weight
        """
        max_weight = -1
        selected_node = self.nodes[0]

        for node in self.nodes:
            weight = self._combined_hash(key, node)
            if weight > max_weight:
                max_weight = weight
                selected_node = node

        return selected_node

    def _combined_hash(self, key: str, node: str) -> int:
        """Combine hash of key and node"""
        # Use SHA256 for good distribution
        combined = f"{key}:{node}"
        hash_obj = hashlib.sha256(combined.encode())
        return int(hash_obj.hexdigest(), 16)

    @staticmethod
    def load_analysis(n_nodes: int, n_items: int) -> Dict:
        """Analyze load distribution for rendezvous hashing"""
        nodes = [f"node{i}" for i in range(n_nodes)]
        hasher = RendezvousHashing(nodes)

        loads = [0] * n_nodes

        for item in range(n_items):
            key = f"item{item}"
            node = hasher.assign(key)
            node_idx = int(node[4:])  # Extract node number
            loads[node_idx] += 1

        mean_load = n_items / n_nodes
        std_load = np.std(loads)
        cv = std_load / mean_load

        # Theoretical: CV ≈ 1/sqrt(mean_load) for good hash functions
        theoretical_cv = 1 / math.sqrt(mean_load)

        return {
            'mean_load': mean_load,
            'actual_cv': cv,
            'theoretical_cv': theoretical_cv,
            'max_load': max(loads),
            'min_load': min(loads),
            'imbalance_ratio': max(loads) / mean_load
        }

class JumpHash:
    """
    Jump Hash: Consistent hashing without virtual nodes

    O(log n) computation, minimal memory, perfect balance
    But can only add/remove nodes at the end
    """

    @staticmethod
    def jump_hash(key: str, num_buckets: int) -> int:
        """
        Assign key to bucket using jump hash algorithm

        Based on: "A Fast, Minimal Memory, Consistent Hash Algorithm"
        by John Lamping and Eric Veach (Google)
        """
        key_hash = ConsistentHashingTheory._hash_position(key, 2**32)

        b = -1
        j = 0

        while j < num_buckets:
            b = j
            key_hash = (key_hash * 2862933555777941757) + 1
            j = int((b + 1) * (2**31 / ((key_hash >> 33) + 1)))

        return b

    @staticmethod
    def analyze_performance(num_buckets_range: List[int], 
                           num_keys: int = 100000) -> List[Dict]:
        """Analyze jump hash performance"""
        results = []

        for num_buckets in num_buckets_range:
            loads = [0] * num_buckets

            for key in range(num_keys):
                bucket = JumpHash.jump_hash(str(key), num_buckets)
                loads[bucket] += 1

            mean_load = num_keys / num_buckets
            std_load = np.std(loads)
            cv = std_load / mean_load

            # Jump hash provides near-perfect balance
            results.append({
                'buckets': num_buckets,
                'mean_load': mean_load,
                'std_load': std_load,
                'coefficient_of_variation': cv,
                'max_load': max(loads),
                'min_load': min(loads),
                'imbalance': max(loads) / mean_load
            })

        return results

    @staticmethod
    def movement_analysis(old_buckets: int, new_buckets: int) -> float:
        """
        Calculate fraction of keys that move when changing bucket count

        Jump hash minimizes movement to approximately 1/new_buckets
        """
        moved = 0
        total = 10000

        for key in range(total):
            old_bucket = JumpHash.jump_hash(str(key), old_buckets)
            new_bucket = JumpHash.jump_hash(str(key), new_buckets)

            if old_bucket != new_bucket:
                moved += 1

        return moved / total

# Comparison of hashing algorithms
class HashingAlgorithmComparison:
    """Compare different consistent hashing algorithms"""

    @staticmethod
    def compare_algorithms(n_nodes: int, 
                          n_items: int,
                          vnodes: int = 100) -> List[Dict]:
        """Compare multiple hashing algorithms"""
        results = []

        # 1. Consistent hashing with virtual nodes
        ch_results = ConsistentHashingTheory.load_distribution(
            n_nodes, n_items, vnodes
        )
        results.append({
            'algorithm': 'consistent_hashing',
            'vnodes': vnodes,
            'cv': ch_results['actual_cv'],
            'imbalance': ch_results['imbalance_ratio'],
            'memory': n_nodes * vnodes,
            'computation': 'O(log(vnodes*n))'
        })

        # 2. Rendezvous hashing
        rh_results = RendezvousHashing.load_analysis(n_nodes, n_items)
        results.append({
            'algorithm': 'rendezvous_hashing',
            'cv': rh_results['actual_cv'],
            'imbalance': rh_results['imbalance_ratio'],
            'memory': 'O(n)',
            'computation': 'O(n)'
        })

        # 3. Jump hash
        jh_results = JumpHash.analyze_performance([n_nodes], n_items)[0]
        results.append({
            'algorithm': 'jump_hash',
            'cv': jh_results['coefficient_of_variation'],
            'imbalance': jh_results['imbalance'],
            'memory': 'O(1)',
            'computation': 'O(log n)',
            'limitation': 'Can only add/remove at end'
        })

        # 4. Maglev hashing (Google's approach)
        # Not implemented here, but known properties
        results.append({
            'algorithm': 'maglev_hashing',
            'cv': 'Near 0',
            'imbalance': '1.0 (perfect)',
            'memory': 'O(n) lookup table',
            'computation': 'O(1)',
            'note': 'Used in Google's load balancers'
        })

        # Sort by imbalance (lower is better)
        results.sort(key=lambda x: x.get('imbalance', float('inf')))

        return results

    @staticmethod
    def recommend_algorithm(requirements: Dict) -> str:
        """Recommend hashing algorithm based on requirements"""
        n_nodes = requirements.get('n_nodes', 10)
        n_items = requirements.get('n_items', 1000000)
        need_perfect_balance = requirements.get('perfect_balance', False)
        need_fast_lookup = requirements.get('fast_lookup', True)
        need_dynamic = requirements.get('dynamic_nodes', True)
        memory_constrained = requirements.get('memory_constrained', False)

        if need_perfect_balance and need_fast_lookup and not memory_constrained:
            return "maglev_hashing"
        elif memory_constrained and not need_dynamic:
            return "jump_hash"
        elif need_fast_lookup and n_nodes <= 1000:
            return "consistent_hashing_with_vnodes"
        elif n_nodes <= 100 and need_perfect_balance:
            return "rendezvous_hashing"
        else:
            return "consistent_hashing_with_vnodes"

# Practical implementation with theoretical guarantees
class TheoreticalConsistentHash:
    """
    Consistent hashing with theoretical guarantees

    Implements algorithm with provable bounds on:
    1. Load imbalance
    2. Movement cost
    3. Lookup time
    """

    def __init__(self, nodes: List[str], vnodes_per_node: int = 100):
        self.nodes = nodes
        self.vnodes_per_node = vnodes_per_node
        self.ring_size = 2**32

        # Build ring with provable properties
        self.ring, self.node_map = self._build_ring()

        # Statistics for theoretical analysis
        self.stats = {
            'lookups': 0,
            'movements': 0,
            'loads': {node: 0 for node in nodes}
        }

    def _build_ring(self) -> Tuple[List[int], Dict[int, str]]:
        """Build ring with theoretical guarantees"""
        ring = []
        node_map = {}

        for node in self.nodes:
            # Use deterministic virtual node positions
            for vnode in range(self.vnodes_per_node):
                # Use SHA256 for uniform distribution
                key = f"{node}:{vnode}"
                hash_bytes = hashlib.sha256(key.encode()).digest()

                # Use first 4 bytes for position
                position = int.from_bytes(hash_bytes[:4], byteorder='big')
                position = position % self.ring_size

                ring.append(position)
                node_map[position] = node

        ring.sort()
        return ring, node_map

    def get_node(self, key: str) -> str:
        """Get node for key with O(log n) lookup"""
        self.stats['lookups'] += 1

        # Hash key
        hash_bytes = hashlib.sha256(key.encode()).digest()
        key_pos = int.from_bytes(hash_bytes[:4], byteorder='big')
        key_pos = key_pos % self.ring_size

        # Binary search in ring
        idx = bisect.bisect_left(self.ring, key_pos)
        if idx == len(self.ring):
            idx = 0

        position = self.ring[idx]
        node = self.node_map[position]

        # Update load statistics
        self.stats['loads'][node] += 1

        return node

    def add_node(self, new_node: str):
        """Add node with minimal movement"""
        old_loads = self.stats['loads'].copy()

        # Add new node to ring
        self.nodes.append(new_node)
        self.stats['loads'][new_node] = 0

        # Rebuild ring (in practice, incremental update)
        self.ring, self.node_map = self._build_ring()

        # Calculate movements
        movements = 0
        # In practice, would track which keys moved

        self.stats['movements'] += movements

    def remove_node(self, node_to_remove: str):
        """Remove node with controlled movement"""
        if node_to_remove not in self.nodes:
            return

        old_loads = self.stats['loads'].copy()

        # Remove node
        self.nodes.remove(node_to_remove)
        del self.stats['loads'][node_to_remove]

        # Rebuild ring
        self.ring, self.node_map = self._build_ring()

        # Redistribute load
        total_redistributed = old_loads[node_to_remove]
        avg_redistribute = total_redistributed / len(self.nodes)

        for node in self.nodes:
            self.stats['loads'][node] += avg_redistribute

    def get_theoretical_bounds(self) -> Dict:
        """Get theoretical performance bounds"""
        n = len(self.nodes)
        k = self.vnodes_per_node

        # Load imbalance bound
        # With k virtual nodes, max load ≤ (1 + ε) * average load
        # where ε = O(1/√k)
        epsilon = 1 / math.sqrt(k)
        max_load_bound = (1 + epsilon) * (sum(self.stats['loads'].values()) / n)

        # Movement bound when adding/removing node
        # Fraction moved ≤ O(1/n)
        movement_bound = 1 / n

        # Lookup time
        lookup_time = math.log2(n * k)  # Binary search in ring

        return {
            'max_load_bound': max_load_bound,
            'actual_max_load': max(self.stats['loads'].values()),
            'movement_bound_per_node_change': movement_bound,
            'lookup_time_complexity': f'O(log({n * k})) = {lookup_time:.1f}',
            'memory_usage': f'{n * k} entries',
            'load_imbalance_ratio': max(self.stats['loads'].values()) / 
                                   (sum(self.stats['loads'].values()) / n)
        }

    def verify_bounds(self, num_tests: int = 10000) -> Dict:
        """Verify theoretical bounds through simulation"""
        # Test load distribution
        test_loads = {node: 0 for node in self.nodes}

        for i in range(num_tests):
            key = f"test_key_{i}"
            node = self.get_node(key)
            test_loads[node] += 1

        actual_max = max(test_loads.values())
        average = num_tests / len(self.nodes)
        actual_imbalance = actual_max / average

        bounds = self.get_theoretical_bounds()
        theoretical_imbalance = bounds['max_load_bound'] / average

        return {
            'actual_imbalance_ratio': actual_imbalance,
            'theoretical_imbalance_bound': theoretical_imbalance,
            'within_bounds': actual_imbalance <= theoretical_imbalance,
            'actual_max_load': actual_max,
            'average_load': average
        }

Consistent Hashing Theory Summary:

Load Balance: With k virtual nodes, imbalance ≤ O(1/√k)
Movement Cost: Adding/removing node moves O(1/n) items
Lookup Time: O(log n) with binary search
Memory: O(nk) for n nodes, k virtual nodes

Algorithm Comparison:

Algorithm	Imbalance	Lookup	Memory	Dynamic
Consistent Hashing	O(1/√k)	O(log n)	O(nk)	✓
Rendezvous	O(1/√n)	O(n)	O(1)	✓
Jump Hash	Perfect	O(log n)	O(1)	✗
Maglev	Perfect	O(1)	O(n)	Limited

📈 16. The Tail at Scale: Why p99 and p999 Matter

Understanding and managing latency outliers in distributed systems.

import numpy as np
from typing import List, Dict, Tuple
import math
from scipy import stats
import random

class TailLatencyAnalysis:
    """Analysis of tail latency in distributed systems"""

    @staticmethod
    def why_tails_matter(mean_latency: float, 
                        std_latency: float,
                        distribution: str = 'normal') -> Dict:
        """
        Demonstrate why p99, p999 matter more than mean

        In distributed systems, user experience is determined
        by worst-case latency, not average
        """
        if distribution == 'normal':
            # For normal distribution
            p50 = mean_latency
            p90 = mean_latency + 1.28 * std_latency
            p95 = mean_latency + 1.645 * std_latency
            p99 = mean_latency + 2.326 * std_latency
            p999 = mean_latency + 3.09 * std_latency
        elif distribution == 'exponential':
            # For exponential distribution (common in queues)
            # p(X > x) = e^(-λx), where λ = 1/mean
            rate = 1 / mean_latency
            p50 = -math.log(0.5) / rate
            p90 = -math.log(0.1) / rate
            p95 = -math.log(0.05) / rate
            p99 = -math.log(0.01) / rate
            p999 = -math.log(0.001) / rate
        else:
            raise ValueError(f"Unknown distribution: {distribution}")

        # Amplification factors
        p90_factor = p90 / mean_latency
        p99_factor = p99 / mean_latency
        p999_factor = p999 / mean_latency

        return {
            'mean': mean_latency,
            'p50': p50,
            'p90': p90,
            'p95': p95,
            'p99': p99,
            'p999': p999,
            'p90_amplification': p90_factor,
            'p99_amplification': p99_factor,
            'p999_amplification': p999_factor,
            'distribution': distribution
        }

    @staticmethod
    def fanout_amplification(service_latency: Dict,
                            fanout: int,
                            parallelism: int) -> Dict:
        """
        Calculate how tail latency amplifies with fanout

        When service calls multiple backends, overall latency
        determined by slowest backend
        """
        mean = service_latency['mean']
        std = service_latency.get('std', mean * 0.5)  # Default CV=0.5

        # Simulate fanout
        n_simulations = 10000
        overall_latencies = []

        for _ in range(n_simulations):
            # Simulate backend latencies
            backend_latencies = np.random.normal(mean, std, fanout)

            # Process in parallel batches
            batch_size = parallelism
            batch_times = []

            for i in range(0, fanout, batch_size):
                batch = backend_latencies[i:i+batch_size]
                batch_time = max(batch)  # Parallel, limited by slowest
                batch_times.append(batch_time)

            # Sequential batches
            total_time = sum(batch_times)
            overall_latencies.append(total_time)

        # Calculate percentiles
        overall_latencies = np.array(overall_latencies)

        return {
            'fanout': fanout,
            'parallelism': parallelism,
            'mean_overall': np.mean(overall_latencies),
            'p50_overall': np.percentile(overall_latencies, 50),
            'p90_overall': np.percentile(overall_latencies, 90),
            'p99_overall': np.percentile(overall_latencies, 99),
            'p999_overall': np.percentile(overall_latencies, 99.9),
            'amplification_p99': np.percentile(overall_latencies, 99) / mean,
            'expected_parallel': mean * math.ceil(fanout / parallelism),
            'actual_vs_expected_ratio': np.mean(overall_latencies) / 
                                      (mean * math.ceil(fanout / parallelism))
        }

    @staticmethod
    def coordinated_omission(measurement_interval: float,
                           service_time_mean: float,
                           service_time_std: float,
                           think_time_mean: float) -> Dict:
        """
        Demonstrate coordinated omission problem

        When system is busy, measurements miss the true latency
        because requests are delayed before even starting
        """
        # Simulate requests over time
        simulation_time = 1000
        current_time = 0
        request_times = []
        start_times = []
        end_times = []

        while current_time < simulation_time:
            # Request arrives
            arrival = current_time
            start_times.append(arrival)

            # Service time (variable)
            service_time = max(0, np.random.normal(service_time_mean, service_time_std))

            # Actual completion
            completion = arrival + service_time
            end_times.append(completion)
            request_times.append(service_time)

            # Next request after think time
            think_time = np.random.exponential(think_time_mean)
            current_time = max(completion, current_time + think_time)

        # Naive measurement (only service time)
        naive_mean = np.mean(request_times)
        naive_p99 = np.percentile(request_times, 99)

        # Correct measurement (including queueing delay)
        queueing_delays = []
        for i in range(1, len(start_times)):
            previous_end = end_times[i-1]
            current_start = start_times[i]

            if current_start < previous_end:
                # Request had to wait
                queueing_delays.append(previous_end - current_start)
            else:
                queueing_delays.append(0)

        # Add queueing to service time
        total_times = []
        for i in range(len(request_times)):
            total_time = request_times[i]
            if i < len(queueing_delays):
                total_time += queueing_delays[i]
            total_times.append(total_time)

        correct_mean = np.mean(total_times)
        correct_p99 = np.percentile(total_times, 99)

        return {
            'naive_mean': naive_mean,
            'naive_p99': naive_p99,
            'correct_mean': correct_mean,
            'correct_p99': correct_p99,
            'omission_error_mean': (correct_mean - naive_mean) / naive_mean,
            'omission_error_p99': (correct_p99 - naive_p99) / naive_p99,
            'queueing_delay_mean': np.mean(queueing_delays),
            'max_queueing_delay': max(queueing_delays) if queueing_delays else 0
        }

class TailReductionTechniques:
    """Techniques for reducing tail latency"""

    @staticmethod
    def hedging_requests(service_latency_mean: float,
                        service_latency_std: float,
                        hedge_delay: float,
                        n_requests: int = 10000) -> Dict:
        """
        Hedge requests: Send duplicate request after delay

        Reduces tail latency at cost of extra load
        """
        primary_times = np.random.normal(service_latency_mean, 
                                        service_latency_std, 
                                        n_requests)

        hedge_times = []
        for i in range(n_requests):
            # Send hedge after delay
            hedge_start = hedge_delay
            hedge_service = np.random.normal(service_latency_mean, 
                                           service_latency_std)
            hedge_completion = hedge_start + hedge_service

            # Take whichever finishes first
            completion = min(primary_times[i], hedge_completion)
            hedge_times.append(completion)

        hedge_times = np.array(hedge_times)

        improvement_p99 = (np.percentile(primary_times, 99) - 
                          np.percentile(hedge_times, 99))
        improvement_p999 = (np.percentile(primary_times, 99.9) - 
                           np.percentile(hedge_times, 99.9))

        # Cost: extra requests sent
        hedge_fraction = sum(hedge_times < primary_times) / n_requests

        return {
            'hedge_delay': hedge_delay,
            'original_p99': np.percentile(primary_times, 99),
            'hedged_p99': np.percentile(hedge_times, 99),
            'original_p999': np.percentile(primary_times, 99.9),
            'hedged_p999': np.percentile(hedge_times, 99.9),
            'p99_improvement': improvement_p99,
            'p999_improvement': improvement_p999,
            'hedge_used_fraction': hedge_fraction,
            'extra_load': hedge_fraction * 100  # Percentage extra requests
        }

    @staticmethod
    def canary_routing(traffic_fraction: float,
                      fast_pool_mean: float,
                      slow_pool_mean: float,
                      std: float) -> Dict:
        """
        Canary routing: Send fraction to different pools

        Helps identify and avoid slow instances
        """
        n_requests = 10000
        canary_count = int(n_requests * traffic_fraction)
        main_count = n_requests - canary_count

        # Generate latencies
        main_latencies = np.random.normal(fast_pool_mean, std, main_count)
        canary_latencies = np.random.normal(slow_pool_mean, std, canary_count)

        # Combine
        all_latencies = np.concatenate([main_latencies, canary_latencies])

        # With perfect detection (avoid slow pool)
        # In reality, detection has some delay
        perfect_latencies = np.random.normal(fast_pool_mean, std, n_requests)

        return {
            'traffic_fraction_to_slow': traffic_fraction,
            'with_canary_p99': np.percentile(all_latencies, 99),
            'perfect_avoidance_p99': np.percentile(perfect_latencies, 99),
            'improvement_if_perfect': (np.percentile(all_latencies, 99) - 
                                      np.percentile(perfect_latencies, 99)),
            'slow_pool_detection_delay': 'Simulated separately'
        }

    @staticmethod
    def latency_injection_detection(true_latency_mean: float,
                                   true_latency_std: float,
                                   injection_prob: float,
                                   injection_amount: float) -> Dict:
        """
        Detect and mitigate latency injections

        Some requests randomly take much longer (GC, compaction, etc.)
        """
        n_requests = 10000
        latencies = []

        for _ in range(n_requests):
            base_latency = np.random.normal(true_latency_mean, true_latency_std)

            # Random injection
            if random.random() < injection_prob:
                latency = base_latency + injection_amount
            else:
                latency = base_latency

            latencies.append(latency)

        latencies = np.array(latencies)

        # Mitigation: timeout and retry
        timeout = np.percentile(latencies, 95)  # 95th percentile as timeout

        mitigated_latencies = []
        retries = 0

        for latency in latencies:
            if latency > timeout:
                # Retry once
                retry_latency = np.random.normal(true_latency_mean, true_latency_std)
                total_latency = timeout + retry_latency  # Wait timeout then retry
                retries += 1
            else:
                total_latency = latency

            mitigated_latencies.append(total_latency)

        mitigated_latencies = np.array(mitigated_latencies)

        return {
            'injection_probability': injection_prob,
            'injection_amount': injection_amount,
            'original_p99': np.percentile(latencies, 99),
            'mitigated_p99': np.percentile(mitigated_latencies, 99),
            'improvement': np.percentile(latencies, 99) - np.percentile(mitigated_latencies, 99),
            'retry_rate': retries / n_requests,
            'timeout_used': timeout,
            'cost_extra_load': retries / n_requests
        }

class StatisticalAnalysis:
    """Statistical tools for tail analysis"""

    @staticmethod
    def extreme_value_theory(n_samples: int,
                           distribution: str = 'normal',
                           block_size: int = 100) -> Dict:
        """
        Apply Extreme Value Theory to model tail behavior

        EVT models distribution of maximum values
        """
        if distribution == 'normal':
            samples = np.random.normal(0, 1, n_samples)
        elif distribution == 'exponential':
            samples = np.random.exponential(1, n_samples)
        else:
            raise ValueError(f"Unsupported distribution: {distribution}")

        # Block maxima approach
        n_blocks = n_samples // block_size
        block_maxima = []

        for i in range(n_blocks):
            block = samples[i*block_size:(i+1)*block_size]
            block_maxima.append(max(block))

        # Fit GEV (Generalized Extreme Value) distribution
        # Simplified: Use Gumbel distribution (Type I EVT)
        block_maxima = np.array(block_maxima)

        # Method of moments for Gumbel
        euler_mascheroni = 0.5772156649
        mean_max = np.mean(block_maxima)
        std_max = np.std(block_maxima)

        # Gumbel parameters
        beta = std_max * math.sqrt(6) / math.pi
        mu = mean_max - euler_mascheroni * beta

        # Predict extreme percentiles
        def gumbel_cdf(x, mu, beta):
            return math.exp(-math.exp(-(x - mu) / beta))

        # Find x such that CDF(x) = p
        def gumbel_quantile(p, mu, beta):
            return mu - beta * math.log(-math.log(p))

        p9999 = gumbel_quantile(0.9999, mu, beta)
        p99999 = gumbel_quantile(0.99999, mu, beta)

        return {
            'distribution': distribution,
            'block_size': block_size,
            'n_blocks': n_blocks,
            'gumbel_mu': mu,
            'gumbel_beta': beta,
            'predicted_p9999': p9999,
            'predicted_p99999': p99999,
            'empirical_max': max(samples),
            'evt_applicable': True if n_blocks >= 30 else False
        }

    @staticmethod
    def heavy_tail_detection(samples: np.ndarray) -> Dict:
        """
        Detect heavy-tailed distributions

        Heavy tails cause extreme outliers that dominate p99, p999
        """
        # Calculate moments
        mean = np.mean(samples)
        std = np.std(samples)
        skew = stats.skew(samples)
        kurtosis = stats.kurtosis(samples)  # Excess kurtosis

        # Hill estimator for tail index
        sorted_samples = np.sort(samples)[::-1]  # Descending
        k = min(100, len(samples) // 10)  # Number of largest values to use

        if k > 1:
            hill_estimator = 1 / (np.mean(np.log(sorted_samples[:k])) - 
                                 np.log(sorted_samples[k]))
        else:
            hill_estimator = float('inf')

        # Check for heavy tail (kurtosis > 3, or tail index < 3)
        is_heavy_tailed = kurtosis > 3 or hill_estimator < 3

        # Compare percentiles
        p50 = np.percentile(samples, 50)
        p99 = np.percentile(samples, 99)
        p999 = np.percentile(samples, 99.9)

        tail_ratio_99 = p99 / p50
        tail_ratio_999 = p999 / p50

        return {
            'mean': mean,
            'std': std,
            'skewness': skew,
            'kurtosis': kurtosis,
            'hill_estimator': hill_estimator,
            'is_heavy_tailed': is_heavy_tailed,
            'p50': p50,
            'p99': p99,
            'p999': p999,
            'tail_ratio_99': tail_ratio_99,
            'tail_ratio_999': tail_ratio_999,
            'heavy_tail_severity': 'severe' if tail_ratio_999 > 10 else 
                                 'moderate' if tail_ratio_999 > 5 else 
                                 'light'
        }

# Practical guidance
class TailLatencyGuidelines:
    """Practical guidelines for managing tail latency"""

    @staticmethod
    def design_for_tail():
        """Design principles for tail latency"""
        return [
            "1. Measure p99, p999, not just average",
            "2. Use hedged requests for idempotent operations",
            "3. Implement canary routing and load shedding",
            "4. Set aggressive timeouts and retry with backoff",
            "5. Use load balancing with health checks",
            "6. Implement circuit breakers for failing dependencies",
            "7. Use queue management and admission control",
            "8. Monitor and alert on latency outliers"
        ]

    @staticmethod
    def monitoring_recommendations():
        """Monitoring recommendations for tail latency"""
        return {
            'metrics_to_track': [
                'Histograms (not averages)',
                'Percentiles: p50, p90, p95, p99, p999',
                'Error rates at different percentiles',
                'Timeouts and retry counts',
                'Queue lengths and wait times'
            ],
            'alerting_thresholds': [
                'p99 > SLA * 2',
                'p999 > SLA * 3',
                'Error rate at p99 > 1%',
                'Consecutive timeouts > 5'
            ],
            'analysis_tools': [
                'Heatmaps over time',
                'Correlation with other metrics',
                'Comparative analysis (A/B testing)',
                'Root cause analysis for outliers'
            ]
        }

    @staticmethod
    def capacity_planning_tail(sla_p99: float,
                              sla_p999: float,
                              current_p99: float,
                              current_p999: float,
                              growth_rate: float) -> Dict:
        """Capacity planning considering tail latency"""

        # Tail latency often scales super-linearly with load
        # Assume p99 ∝ load^α, where α > 1

        # Estimate α from current measurements
        # Simplified: α = log(p99_high_load / p99_low_load) / log(load_ratio)
        # In practice, measure at different load levels

        # For planning, use conservative α = 1.5 (common for queueing systems)
        alpha = 1.5

        # Current utilization (estimated)
        current_utilization = 0.7  # 70% utilization

        # Load at which SLA would be violated
        def load_at_violation(current_load, current_latency, target_latency, alpha):
            # latency ∝ load^α
            # target_latency / current_latency = (target_load / current_load)^α
            ratio = target_latency / current_latency
            target_load = current_load * (ratio ** (1/alpha))
            return target_load

        load_violation_p99 = load_at_violation(
            current_utilization, current_p99, sla_p99, alpha
        )

        load_violation_p999 = load_at_violation(
            current_utilization, current_p999, sla_p999, alpha
        )

        # Which violates first
        first_violation_load = min(load_violation_p99, load_violation_p999)
        violating_metric = 'p99' if load_violation_p99 < load_violation_p999 else 'p999'

        # Time to violation with growth
        time_to_violation = math.log(first_violation_load / current_utilization) / math.log(1 + growth_rate)

        return {
            'alpha_estimated': alpha,
            'current_utilization': current_utilization,
            'load_at_p99_violation': load_violation_p99,
            'load_at_p999_violation': load_violation_p999,
            'first_violation_at_load': first_violation_load,
            'violating_metric': violating_metric,
            'time_to_violation_months': time_to_violation * 12,  # Convert to months if growth_rate monthly
            'recommendation': f'Plan capacity upgrade within {time_to_violation:.1f} growth periods'
        }

Tail at Scale Key Insights:

Percentiles Matter: p99 is 10-100x more important than mean for user experience
Fanout Amplification: Calling N services → latency = max(service latencies)
Coordinated Omission: Busy systems hide true latency in queueing delays
Heavy Tails: Real-world latency distributions often have heavy tails
Mitigation Techniques: Hedging, canary routing, timeouts, retries

Empirical Findings from Google:

p99 latency often 10x p50
p999 latency often 100x p50
Fanout of 100 servers → p99 = p99 of single server ^ 100
Hedging with 1ms delay reduces p99 by 2x with 1% extra load

🌉 17. Network Calculus: Deterministic Performance Bounds

Mathematical framework for guaranteed performance in networks.

import numpy as np
from typing import List, Tuple, Callable
import math

class ArrivalCurve:
    """Arrival curve α(t) bounds arrivals in any interval"""

    def __init__(self, rate: float, burst: float):
        """
        α(t) = rate * t + burst

        rate: long-term arrival rate
        burst: maximum burst size
        """
        self.rate = rate
        self.burst = burst

    def bound(self, interval: float) -> float:
        """Maximum arrivals in time interval t"""
        return self.rate * interval + self.burst

    def is_conformant(self, arrivals: List[Tuple[float, float]]) -> bool:
        """
        Check if sequence of (time, amount) conforms to curve

        For all intervals [s, t], arrivals in interval ≤ α(t-s)
        """
        n = len(arrivals)

        for i in range(n):
            for j in range(i, n):
                start_time, _ = arrivals[i]
                end_time, _ = arrivals[j]
                interval = end_time - start_time

                # Sum arrivals in [start_time, end_time]
                total = sum(amount for time, amount in arrivals[i:j+1] 
                           if start_time <= time <= end_time)

                if total > self.bound(interval):
                    return False

        return True

class ServiceCurve:
    """Service curve β(t) bounds service provided"""

    def __init__(self, rate: float, latency: float = 0):
        """
        β(t) = max(0, rate * (t - latency))

        rate: service rate
        latency: initial latency before service starts
        """
        self.rate = rate
        self.latency = latency

    def bound(self, interval: float) -> float:
        """Minimum service in time interval t"""
        return max(0, self.rate * (interval - self.latency))

class NetworkCalculus:
    """Network Calculus operations"""

    @staticmethod
    def convolution(alpha: ArrivalCurve, beta: ServiceCurve) -> ArrivalCurve:
        """
        Output bound: α ⊗ β

        After passing through service curve β,
        arrivals are bounded by α ⊗ β
        """
        # For affine curves: (r, b) ⊗ (R, T) = (r, b + r * T)
        # If r ≤ R (stable system)
        if alpha.rate > beta.rate:
            raise ValueError("Unstable: arrival rate > service rate")

        output_rate = alpha.rate
        output_burst = alpha.burst + alpha.rate * beta.latency

        return ArrivalCurve(output_rate, output_burst)

    @staticmethod
    def concatenation(beta1: ServiceCurve, beta2: ServiceCurve) -> ServiceCurve:
        """
        Concatenation of service curves: β1 ⊗ β2

        Service through two nodes in series
        """
        # For rate-latency curves: (R1, T1) ⊗ (R2, T2) = (min(R1, R2), T1 + T2)
        combined_rate = min(beta1.rate, beta2.rate)
        combined_latency = beta1.latency + beta2.latency

        return ServiceCurve(combined_rate, combined_latency)

    @staticmethod
    def backlog_bound(alpha: ArrivalCurve, beta: ServiceCurve) -> float:
        """
        Maximum backlog (queue size) bound

        sup_t [α(t) - β(t)]
        """
        # For affine arrival and rate-latency service:
        # max backlog = b + r * T
        return alpha.burst + alpha.rate * beta.latency

    @staticmethod
    def delay_bound(alpha: ArrivalCurve, beta: ServiceCurve) -> float:
        """
        Maximum delay bound

        Horizontal deviation between α and β
        """
        if alpha.rate > beta.rate:
            return float('inf')  # Unstable

        # For affine arrival and rate-latency service:
        # max delay = T + b / R
        return beta.latency + alpha.burst / beta.rate

    @staticmethod
    def leftover_service(alpha: ArrivalCurve, 
                        beta: ServiceCurve,
                        priority: str = 'strict') -> ServiceCurve:
        """
        Service curve left for other traffic

        After serving traffic with arrival curve α
        """
        if priority == 'strict':
            # Strict priority: β'(t) = max(0, β(t) - α(t))
            # For affine curves, approximate
            leftover_rate = max(0, beta.rate - alpha.rate)
            # Latency increases
            extra_latency = alpha.burst / beta.rate if beta.rate > 0 else float('inf')
            leftover_latency = beta.latency + extra_latency

            return ServiceCurve(leftover_rate, leftover_latency)
        else:
            # Other scheduling policies more complex
            raise NotImplementedError(f"Priority {priority} not implemented")

class DeterministicNetworkAnalysis:
    """Analyze network with deterministic guarantees"""

    def __init__(self):
        self.flows = {}  # flow_id -> ArrivalCurve
        self.nodes = {}  # node_id -> ServiceCurve
        self.routing = {}  # flow_id -> List[node_id]

    def add_flow(self, flow_id: str, arrival_curve: ArrivalCurve):
        self.flows[flow_id] = arrival_curve

    def add_node(self, node_id: str, service_curve: ServiceCurve):
        self.nodes[node_id] = service_curve

    def set_routing(self, flow_id: str, path: List[str]):
        self.routing[flow_id] = path

    def analyze_flow(self, flow_id: str) -> Dict:
        """Analyze end-to-end guarantees for flow"""
        if flow_id not in self.flows:
            raise ValueError(f"Flow {flow_id} not found")
        if flow_id not in self.routing:
            raise ValueError(f"Routing for flow {flow_id} not found")

        arrival = self.flows[flow_id]
        path = self.routing[flow_id]

        # Combine service curves along path
        if not path:
            raise ValueError(f"Empty path for flow {flow_id}")

        # Start with first node
        combined_service = self.nodes[path[0]]

        # Concatenate remaining nodes
        for node_id in path[1:]:
            node_service = self.nodes[node_id]
            combined_service = NetworkCalculus.concatenation(
                combined_service, node_service
            )

        # Calculate bounds
        max_backlog = NetworkCalculus.backlog_bound(arrival, combined_service)
        max_delay = NetworkCalculus.delay_bound(arrival, combined_service)

        # Output arrival curve
        output_arrival = NetworkCalculus.convolution(arrival, combined_service)

        return {
            'flow_id': flow_id,
            'path': path,
            'arrival_curve': (arrival.rate, arrival.burst),
            'service_curve': (combined_service.rate, combined_service.latency),
            'max_backlog': max_backlog,
            'max_delay': max_delay,
            'output_curve': (output_arrival.rate, output_arrival.burst),
            'stable': arrival.rate <= combined_service.rate
        }

    def analyze_node(self, node_id: str) -> Dict:
        """Analyze node utilization and leftover service"""
        if node_id not in self.nodes:
            raise ValueError(f"Node {node_id} not found")

        node_service = self.nodes[node_id]

        # Find all flows through this node
        flows_through = []
        total_arrival = ArrivalCurve(0, 0)

        for flow_id, path in self.routing.items():
            if node_id in path:
                flows_through.append(flow_id)
                flow_arrival = self.flows[flow_id]

                # Sum arrival curves (assuming independence)
                total_arrival.rate += flow_arrival.rate
                total_arrival.burst += flow_arrival.burst

        # Calculate utilization
        utilization = total_arrival.rate / node_service.rate if node_service.rate > 0 else float('inf')

        # Leftover service
        leftover = NetworkCalculus.leftover_service(total_arrival, node_service)

        return {
            'node_id': node_id,
            'service_curve': (node_service.rate, node_service.latency),
            'flows_through': flows_through,
            'total_arrival': (total_arrival.rate, total_arrival.burst),
            'utilization': utilization,
            'leftover_service': (leftover.rate, leftover.latency),
            'stable': total_arrival.rate <= node_service.rate
        }

class TSN_Analysis:
    """Time-Sensitive Networking analysis using Network Calculus"""

    @staticmethod
    def time_aware_shaping(cycle_time: float,
                          slot_duration: float,
                          frame_sizes: List[float],
                          link_speed: float) -> Dict:
        """
        Analyze Time-Aware Shaping (802.1Qbv)

        Cycle repeats every cycle_time
        Slots within cycle for different traffic classes
        """
        n_slots = int(cycle_time / slot_duration)

        # Service curve for a traffic class with dedicated slot
        def service_curve_for_slot(slot_index: int, frame_size: float) -> ServiceCurve:
            """
            Service during specific slot each cycle

            Simplified: Can send one frame per cycle
            More accurate: Account for transmission time
            """
            # Minimum service: one frame per cycle
            min_rate = frame_size / cycle_time

            # Maximum latency: wait entire cycle
            max_latency = cycle_time

            return ServiceCurve(min_rate, max_latency)

        results = []
        for i, frame_size in enumerate(frame_sizes):
            slot = i % n_slots
            service = service_curve_for_slot(slot, frame_size)

            # Maximum delay for flow using this slot
            # Assuming worst-case arrival right after slot
            max_delay = cycle_time - slot_duration + frame_size / link_speed

            results.append({
                'slot': slot,
                'frame_size': frame_size,
                'guaranteed_rate': service.rate,
                'max_latency': service.latency,
                'max_delay_with_transmission': max_delay,
                'cycle_time': cycle_time,
                'slot_duration': slot_duration
            })

        return results

    @staticmethod
    def credit_based_shaper(critical_bandwidth: float,
                           idle_slope: float,
                           send_slope: float,
                           max_frame_size: float) -> Dict:
        """
        Analyze Credit-Based Shaper (802.1Qav)

        Used for Audio Video Bridging (AVB)
        """
        # Credit increases at idle_slope when not sending
        # Decreases at send_slope when sending
        # Bounded by [0, maxCredit]

        max_credit = max_frame_size * (idle_slope / send_slope)

        # Service curve approximation
        # Can send when credit >= frame_size
        # Credit builds at rate idle_slope

        # Minimum service rate
        min_rate = idle_slope

        # Maximum burst: can send max_credit worth of data at once
        max_burst = max_credit

        # Latency: time to build enough credit for first frame
        latency = max_frame_size / idle_slope

        service = ServiceCurve(min_rate, latency)

        return {
            'idle_slope': idle_slope,
            'send_slope': send_slope,
            'max_credit': max_credit,
            'max_frame_size': max_frame_size,
            'service_curve_rate': service.rate,
            'service_curve_latency': service.latency,
            'max_burst': max_burst,
            'notes': 'Simplified analysis, actual CBS more complex'
        }

# Practical application: Industrial Ethernet
class IndustrialNetworkDesign:
    """Design industrial networks with deterministic guarantees"""

    def __init__(self):
        self.machines = {}
        self.switches = {}
        self.flows = {}

    def add_machine(self, machine_id: str, 
                   cycle_time: float,
                   data_per_cycle: float):
        """Add a machine with periodic traffic"""
        self.machines[machine_id] = {
            'cycle_time': cycle_time,
            'data_per_cycle': data_per_cycle,
            'arrival_curve': ArrivalCurve(
                rate=data_per_cycle / cycle_time,
                burst=data_per_cycle  # One cycle's worth
            )
        }

    def add_switch(self, switch_id: str, speed: float):
        """Add a switch with given speed"""
        # Simple service curve: rate = speed, latency = transmission delay
        # For accurate analysis, include processing delay
        processing_delay = 0.000010  # 10 microseconds
        self.switches[switch_id] = ServiceCurve(speed, processing_delay)

    def add_flow(self, flow_id: str, 
                source: str, 
                destination: str,
                path: List[str],
                priority: int = 0):
        """Add a data flow"""
        if source not in self.machines:
            raise ValueError(f"Source machine {source} not found")

        arrival = self.machines[source]['arrival_curve']

        self.flows[flow_id] = {
            'source': source,
            'destination': destination,
            'path': path,
            'priority': priority,
            'arrival': arrival
        }

    def verify_schedulability(self) -> Dict:
        """Verify all flows meet deadlines"""
        results = {
            'feasible': True,
            'violations': [],
            'flow_analysis': []
        }

        for flow_id, flow in self.flows.items():
            arrival = flow['arrival']
            path = flow['path']

            # Combine service along path
            if not path:
                continue

            # Check each switch in path exists
            for switch_id in path:
                if switch_id not in self.switches:
                    results['feasible'] = False
                    results['violations'].append(
                        f"Flow {flow_id}: Switch {switch_id} not found"
                    )
                    break

            if not results['feasible']:
                continue

            # Combine service curves
            combined_service = self.switches[path[0]]
            for switch_id in path[1:]:
                combined_service = NetworkCalculus.concatenation(
                    combined_service, self.switches[switch_id]
                )

            # Calculate delay
            delay = NetworkCalculus.delay_bound(arrival, combined_service)

            # For industrial systems, deadline is typically cycle time
            source_machine = self.machines[flow['source']]
            deadline = source_machine['cycle_time']

            meets_deadline = delay <= deadline

            if not meets_deadline:
                results['feasible'] = False
                results['violations'].append(
                    f"Flow {flow_id}: Delay {delay:.6f}s > Deadline {deadline:.6f}s"
                )

            results['flow_analysis'].append({
                'flow_id': flow_id,
                'path': path,
                'arrival_rate': arrival.rate,
                'service_rate': combined_service.rate,
                'calculated_delay': delay,
                'deadline': deadline,
                'meets_deadline': meets_deadline,
                'utilization': arrival.rate / combined_service.rate
            })

        return results

    def calculate_network_utilization(self) -> Dict:
        """Calculate utilization of each network element"""
        utilizations = {}

        for switch_id, service in self.switches.items():
            total_rate = 0

            # Sum rates of all flows through this switch
            for flow_id, flow in self.flows.items():
                if switch_id in flow['path']:
                    total_rate += flow['arrival'].rate

            utilization = total_rate / service.rate
            utilizations[switch_id] = {
                'total_rate': total_rate,
                'service_rate': service.rate,
                'utilization': utilization,
                'stable': total_rate <= service.rate
            }

        return utilizations

Network Calculus Key Results:

Arrival Curve α(t): Bounds traffic arrivals
Service Curve β(t): Bounds service provided
Backlog Bound: sup[α(t) - β(t)]
Delay Bound: Horizontal deviation between α and β
Concatenation: β1 ⊗ β2 for series nodes
Leftover Service: β' = β - α for remaining capacity

Applications:

TSN/AVB: Time-sensitive networking standards
Industrial Ethernet: Factory automation
5G URLLC: Ultra-reliable low-latency communication
Real-time systems: Hard deadline guarantees

💧 18. Fluid Flow Models: Continuous Traffic Approximation

Modeling traffic as continuous flow rather than discrete packets.

import numpy as np
from scipy import integrate, optimize
from typing import Callable, List, Tuple
import math

class FluidFlowModel:
    """Continuous fluid approximation of network traffic"""

    def __init__(self, initial_state: float = 0):
        self.state = initial_state  # Queue length or buffer occupancy
        self.time = 0

    def ode_system(self, t: float, y: float, 
                  arrival_rate: Callable[[float], float],
                  service_rate: Callable[[float], float]) -> float:
        """
        Ordinary Differential Equation for queue dynamics

        dy/dt = λ(t) - μ(t)  for y > 0
        dy/dt = max(0, λ(t) - μ(t))  for y = 0
        """
        lambda_t = arrival_rate(t)
        mu_t = service_rate(t)

        if y > 0:
            return lambda_t - mu_t
        else:
            return max(0, lambda_t - mu_t)

    def simulate(self, 
                arrival_rate: Callable[[float], float],
                service_rate: Callable[[float], float],
                duration: float,
                dt: float = 0.01) -> Tuple[np.ndarray, np.ndarray]:
        """
        Simulate fluid queue using Euler method

        Returns: (time_points, queue_lengths)
        """
        n_steps = int(duration / dt)
        times = np.linspace(0, duration, n_steps)
        queues = np.zeros(n_steps)

        current_queue = self.state

        for i, t in enumerate(times):
            # Update queue
            if current_queue > 0:
                dq_dt = arrival_rate(t) - service_rate(t)
            else:
                dq_dt = max(0, arrival_rate(t) - service_rate(t))

            current_queue += dq_dt * dt
            current_queue = max(0, current_queue)  # Non-negative

            queues[i] = current_queue

        return times, queues

    def steady_state_analysis(self, 
                            arrival_rate: float,
                            service_rate: float) -> Dict:
        """
        Steady-state analysis of fluid queue

        For constant rates λ and μ
        """
        if arrival_rate >= service_rate:
            # Unstable, queue grows without bound
            return {
                'stable': False,
                'utilization': arrival_rate / service_rate,
                'mean_queue': float('inf'),
                'mean_delay': float('inf')
            }

        # M/M/1 fluid approximation
        rho = arrival_rate / service_rate

        # For fluid model with exponential assumptions
        mean_queue = rho / (1 - rho)
        mean_delay = mean_queue / arrival_rate

        # Distribution (approximate)
        def queue_distribution(q: float) -> float:
            # P(queue > q) = e^{-(μ-λ)q}
            return math.exp(-(service_rate - arrival_rate) * q)

        return {
            'stable': True,
            'arrival_rate': arrival_rate,
            'service_rate': service_rate,
            'utilization': rho,
            'mean_queue': mean_queue,
            'mean_delay': mean_delay,
            'p_queue_exceeds': queue_distribution,
            'p_queue_exceeds_1': queue_distribution(1),
            'p_queue_exceeds_10': queue_distribution(10)
        }

class NetworkFluidModel:
    """Fluid model of network with multiple queues"""

    def __init__(self, n_queues: int):
        self.n_queues = n_queues
        self.queues = [0] * n_queues
        self.routing = np.eye(n_queues)  # Default: no routing between queues

    def set_routing(self, routing_matrix: np.ndarray):
        """
        routing_matrix[i][j] = fraction from queue i to queue j
        routing_matrix[i][0] = fraction leaving system from i
        """
        if routing_matrix.shape != (self.n_queues, self.n_queues):
            raise ValueError("Routing matrix must be n x n")

        # Rows should sum to 1 (conservation of flow)
        for i in range(self.n_queues):
            if not math.isclose(np.sum(routing_matrix[i]), 1, rel_tol=1e-9):
                raise ValueError(f"Row {i} does not sum to 1")

        self.routing = routing_matrix

    def network_ode(self, t: float, y: np.ndarray,
                   external_arrivals: np.ndarray,
                   service_rates: np.ndarray) -> np.ndarray:
        """
        ODE system for network of fluid queues

        dy_i/dt = external_i + Σ_j y_j * μ_j * r_ji - μ_i * y_i
        where y_i = queue length at node i
        """
        dydt = np.zeros(self.n_queues)

        # Current service rates (could depend on queue lengths)
        current_service = np.minimum(service_rates, y)  # Can't serve more than queue

        # Flow balance
        for i in range(self.n_queues):
            # External arrivals
            inflow = external_arrivals[i]

            # Internal arrivals from other queues
            for j in range(self.n_queues):
                if i != j:
                    inflow += current_service[j] * self.routing[j][i]

            # Outflow (service)
            outflow = current_service[i]

            dydt[i] = inflow - outflow

        return dydt

    def simulate_network(self,
                        external_arrivals: Callable[[float], np.ndarray],
                        service_rates: Callable[[float], np.ndarray],
                        duration: float,
                        dt: float = 0.01) -> Tuple[np.ndarray, List[np.ndarray]]:
        """Simulate network of fluid queues"""
        n_steps = int(duration / dt)
        times = np.linspace(0, duration, n_steps)
        queue_histories = [np.zeros(n_steps) for _ in range(self.n_queues)]

        current_queues = np.array(self.queues, dtype=float)

        for step, t in enumerate(times):
            # Get current rates
            ext_arr = external_arrivals(t)
            svc_rates = service_rates(t)

            # Update using Euler method
            dq_dt = self.network_ode(t, current_queues, ext_arr, svc_rates)
            current_queues += dq_dt * dt
            current_queues = np.maximum(current_queues, 0)  # Non-negative

            # Store
            for i in range(self.n_queues):
                queue_histories[i][step] = current_queues[i]

        return times, queue_histories

    def steady_state_network(self,
                           external_arrivals: np.ndarray,
                           service_rates: np.ndarray) -> Dict:
        """
        Calculate steady-state of network

        Solve: 0 = λ + R^T * μ - μ
        where λ = external arrivals, R = routing matrix
        """
        # Solve linear equations for effective arrival rates
        # Λ = λ + R^T Λ
        # (I - R^T) Λ = λ

        I = np.eye(self.n_queues)
        A = I - self.routing.T

        try:
            effective_arrivals = np.linalg.solve(A, external_arrivals)
        except np.linalg.LinAlgError:
            # Singular matrix, use least squares
            effective_arrivals = np.linalg.lstsq(A, external_arrivals, rcond=None)[0]

        # Check stability: effective_arrivals_i < service_rates_i
        stable = all(effective_arrivals[i] < service_rates[i] 
                    for i in range(self.n_queues))

        if not stable:
            return {
                'stable': False,
                'effective_arrivals': effective_arrivals,
                'service_rates': service_rates,
                'unstable_nodes': [i for i in range(self.n_queues)
                                  if effective_arrivals[i] >= service_rates[i]]
            }

        # Calculate queue lengths (M/M/1 approximation)
        utilizations = effective_arrivals / service_rates
        mean_queues = utilizations / (1 - utilizations)
        mean_delays = mean_queues / effective_arrivals

        # Total network
        total_arrival = np.sum(external_arrivals)
        total_queue = np.sum(mean_queues)
        total_delay = total_queue / total_arrival if total_arrival > 0 else 0

        return {
            'stable': True,
            'effective_arrivals': effective_arrivals.tolist(),
            'utilizations': utilizations.tolist(),
            'mean_queues': mean_queues.tolist(),
            'mean_delays': mean_delays.tolist(),
            'total_arrival': total_arrival,
            'total_queue': total_queue,
            'total_delay': total_delay,
            'bottleneck_node': np.argmax(utilizations)
        }

class TCPFluidModel:
    """Fluid model of TCP congestion control"""

    def __init__(self, 
                 num_flows: int,
                 link_capacity: float,
                 propagation_delay: float = 0.1):
        self.num_flows = num_flows
        self.link_capacity = link_capacity
        self.prop_delay = propagation_delay

        # State variables
        self.window_sizes = [1.0] * num_flows  # Congestion window
        self.queue_length = 0.0
        self.rtts = [2 * propagation_delay] * num_flows

    def tcp_ode(self, t: float, 
                w: List[float],  # window sizes
                q: float) -> Tuple[List[float], float]:
        """
        ODE model of TCP Reno/RED

        Based on: "A Fluid-based Analysis of a Network of AQM Routers
        Supporting TCP Flows" by Misra, Gong, Towsley
        """
        # Calculate total arrival rate
        total_rate = sum(wi / rtti for wi, rtti in zip(w, self.rtts))

        # Queue dynamics
        dq_dt = total_rate - self.link_capacity
        dq_dt = max(dq_dt, -self.link_capacity)  # Can't deplete faster than capacity

        # TCP window dynamics
        dw_dt = []

        for i in range(self.num_flows):
            # Current RTT = propagation + queueing delay
            current_rtt = self.prop_delay + q / self.link_capacity
            self.rtts[i] = current_rtt

            # TCP Reno: increase by 1/RTT per RTT, decrease by w/2 on loss
            # Loss probability based on queue length (simplified RED)
            p_loss = self._red_loss_probability(q)

            # Window update
            dwdt = 1/current_rtt - w[i] * p_loss / (2 * current_rtt)
            dw_dt.append(dwdt)

        return dw_dt, dq_dt

    def _red_loss_probability(self, q: float) -> float:
        """Random Early Detection (RED) loss probability"""
        min_thresh = 5.0
        max_thresh = 15.0
        max_p = 0.1

        if q <= min_thresh:
            return 0.0
        elif q >= max_thresh:
            return max_p
        else:
            return max_p * (q - min_thresh) / (max_thresh - min_thresh)

    def simulate_tcp(self, duration: float, dt: float = 0.01) -> Dict:
        """Simulate TCP dynamics"""
        n_steps = int(duration / dt)
        times = np.linspace(0, duration, n_steps)

        # History
        window_history = [np.zeros(n_steps) for _ in range(self.num_flows)]
        queue_history = np.zeros(n_steps)
        rate_history = [np.zeros(n_steps) for _ in range(self.num_flows)]

        # Initial state
        w = self.window_sizes.copy()
        q = self.queue_length

        for step, t in enumerate(times):
            # Store current state
            for i in range(self.num_flows):
                window_history[i][step] = w[i]
                rate_history[i][step] = w[i] / self.rtts[i] if self.rtts[i] > 0 else 0

            queue_history[step] = q

            # Update using Euler method
            dw_dt, dq_dt = self.tcp_ode(t, w, q)

            for i in range(self.num_flows):
                w[i] += dw_dt[i] * dt
                w[i] = max(w[i], 1.0)  # Minimum window

            q += dq_dt * dt
            q = max(q, 0.0)  # Non-negative queue

        # Statistics
        final_rates = [w[i] / self.rtts[i] for i in range(self.num_flows)]
        total_rate = sum(final_rates)
        utilization = total_rate / self.link_capacity
        avg_queue = np.mean(queue_history)
        avg_rtt = np.mean(self.rtts)

        # Fairness index (Jain's index)
        rates = final_rates
        n = len(rates)
        numerator = sum(rates) ** 2
        denominator = n * sum(r**2 for r in rates)
        fairness = numerator / denominator if denominator > 0 else 0

        return {
            'times': times,
            'windows': window_history,
            'queue': queue_history,
            'rates': rate_history,
            'final_rates': final_rates,
            'total_rate': total_rate,
            'utilization': utilization,
            'average_queue': avg_queue,
            'average_rtt': avg_rtt,
            'fairness_index': fairness,
            'converged': abs(utilization - 1.0) < 0.1  # Close to full utilization
        }

class FluidApproximationAccuracy:
    """Analyze accuracy of fluid approximations"""

    @staticmethod
    def compare_discrete_fluid(arrival_process: str,
                              service_process: str,
                              duration: float,
                              num_samples: int = 1000) -> Dict:
        """
        Compare discrete event simulation vs fluid approximation
        """
        # Parameters
        mean_arrival = 10.0  # packets/second
        mean_service = 12.0  # packets/second

        if arrival_process == 'poisson':
            # Discrete: Poisson arrivals
            interarrival_times = np.random.exponential(1/mean_arrival, num_samples)
            arrival_times = np.cumsum(interarrival_times)
            arrival_times = arrival_times[arrival_times <= duration]
        elif arrival_process == 'constant':
            # Constant rate
            interval = 1 / mean_arrival
            arrival_times = np.arange(0, duration, interval)
        else:
            raise ValueError(f"Unknown arrival process: {arrival_process}")

        if service_process == 'exponential':
            service_times = np.random.exponential(1/mean_service, len(arrival_times))
        elif service_process == 'constant':
            service_times = np.full(len(arrival_times), 1/mean_service)
        else:
            raise ValueError(f"Unknown service process: {service_process}")

        # Discrete event simulation
        departure_times = []
        current_time = 0

        for i, arrival in enumerate(arrival_times):
            start_service = max(arrival, current_time)
            departure = start_service + service_times[i]
            departure_times.append(departure)
            current_time = departure

        # Calculate queue length over time (discrete)
        time_points = np.linspace(0, duration, 1000)
        discrete_queue = np.zeros_like(time_points)

        for i, t in enumerate(time_points):
            arrivals_by_t = np.sum(arrival_times <= t)
            departures_by_t = np.sum(np.array(departure_times) <= t)
            discrete_queue[i] = arrivals_by_t - departures_by_t

        # Fluid approximation
        fluid_model = FluidFlowModel()
        times, fluid_queue = fluid_model.simulate(
            arrival_rate=lambda t: mean_arrival,
            service_rate=lambda t: mean_service,
            duration=duration,
            dt=0.01
        )

        # Interpolate fluid to same time points
        from scipy import interpolate
        if len(times) > 1 and len(fluid_queue) > 1:
            fluid_interp = interpolate.interp1d(times, fluid_queue, 
                                               bounds_error=False, 
                                               fill_value=0)
            fluid_at_points = fluid_interp(time_points)
        else:
            fluid_at_points = np.zeros_like(time_points)

        # Calculate errors
        mae = np.mean(np.abs(discrete_queue - fluid_at_points))
        mse = np.mean((discrete_queue - fluid_at_points) ** 2)
        max_error = np.max(np.abs(discrete_queue - fluid_at_points))

        # Steady-state comparison
        discrete_mean = np.mean(discrete_queue)
        fluid_mean = np.mean(fluid_queue)

        # Theoretical M/M/1
        rho = mean_arrival / mean_service
        theoretical_mean = rho / (1 - rho) if rho < 1 else float('inf')

        return {
            'arrival_process': arrival_process,
            'service_process': service_process,
            'utilization': rho,
            'discrete_mean_queue': discrete_mean,
            'fluid_mean_queue': fluid_mean,
            'theoretical_mean_queue': theoretical_mean,
            'mean_absolute_error': mae,
            'mean_squared_error': mse,
            'max_error': max_error,
            'relative_error_mean': abs(fluid_mean - discrete_mean) / discrete_mean if discrete_mean > 0 else 0,
            'fluid_accuracy': 'good' if mae < 1.0 else 'fair' if mae < 3.0 else 'poor',
            'time_points': time_points,
            'discrete_queue': discrete_queue,
            'fluid_queue': fluid_at_points
        }

    @staticmethod
    def scaling_accuracy(num_flows: int,
                        duration: float = 100) -> List[Dict]:
        """How accuracy improves with scaling (more flows)"""
        results = []

        for n in [1, 2, 5, 10, 20, 50, 100]:
            # Simulate n independent M/M/1 queues
            total_discrete = 0
            total_fluid = 0

            for _ in range(n):
                comparison = FluidApproximationAccuracy.compare_discrete_fluid(
                    'poisson', 'exponential', duration, num_samples=1000
                )
                total_discrete += comparison['discrete_mean_queue']
                total_fluid += comparison['fluid_mean_queue']

            avg_discrete = total_discrete / n
            avg_fluid = total_fluid / n
            error = abs(avg_fluid - avg_discrete) / avg_discrete if avg_discrete > 0 else 0

            results.append({
                'num_flows': n,
                'avg_discrete': avg_discrete,
                'avg_fluid': avg_fluid,
                'relative_error': error,
                'fluid_law_of_large_numbers': error < 1/n  # Should improve with n
            })

        return results

Fluid Flow Model Insights:

Continuous Approximation: Treat traffic as fluid flow rather than discrete packets
ODE Models: Describe system dynamics with differential equations
Scalability: More accurate with many flows (law of large numbers)
Analytical Tractability: Easier to analyze than discrete event simulation
TCP Modeling: Can model congestion control dynamics

Applications:

Large-scale networks: Data centers, backbone networks
Congestion control analysis: TCP variants, AQM schemes
Performance bounds: Worst-case analysis
Control theory: Design of network controllers

🎯 Conclusion: From Theory to Practice

We've journeyed through the deep theoretical foundations of distributed systems. Let's summarize the key takeaways:

class DistributedTheoryCheatSheet:
    """Quick reference for distributed systems theory"""

    @staticmethod
    def when_to_use_what():
        return {
            'clock_sync': {
                'lamport': 'Causal ordering without physical clocks',
                'vector': 'Detect concurrency, more expensive',
                'hybrid': 'Mix physical and logical, good balance',
                'truetime': 'Global consistency (Spanner, requires HW)',
                'itc': 'Dynamic systems, fork/join operations'
            },
            'consensus': {
                'paxos': 'Classic, proven, complex',
                'raft': 'Understandable, production-ready',
                'byzantine': 'Adversarial environments',
                'flp': 'Remember: async + deterministic = impossible'
            },
            'consistency': {
                'linearizable': 'Strongest, banking, locks',
                'sequential': 'Weaker but faster',
                'causal': 'Preserves happens-before',
                'eventual': 'Weak, high availability',
                'red_blue': 'Mix strong and eventual'
            },
            'load_balancing': {
                'power_of_d': 'd=2 gives most benefit',
                'consistent_hashing': 'Minimal movement',
                'rendezvous': 'Perfect balance, O(n) cost',
                'jump_hash': 'Perfect balance, O(log n), static'
            },
            'performance_modeling': {
                'queueing_theory': 'Predict means, steady-state',
                'network_calculus': 'Worst-case bounds',
                'fluid_models': 'Continuous approximation',
                'tail_analysis': 'Focus on p99, p999'
            },
            'concurrency': {
                'lock_based': 'Simple, can deadlock',
                'lock_free': 'Progress guaranteed, individual starvation',
                'wait_free': 'Bounded steps, most expensive',
                'obstruction_free': 'Weakest, progress with no contention'
            }
        }

    @staticmethod
    def practical_advice():
        return [
            "1. Start with Raft, not Paxos (unless you need Byzantine)",
            "2. Use Hybrid Logical Clocks for most applications",
            "3. Power of 2 choices for load balancing",
            "4. Measure p99, p999, not just averages",
            "5. Use queueing theory for capacity planning",
            "6. Consider consistency trade-offs (CAP, PACELC)",
            "7. Implement hedging for tail latency reduction",
            "8. Use network calculus for hard real-time systems",
            "9. Fluid models good for large-scale analysis",
            "10. Theory informs, but measure your actual system"
        ]

    @staticmethod
    def next_steps():
        return {
            'read': [
                "Leslie Lamport's papers",
                "Google's Spanner, Chubby papers",
                "Amazon's Dynamo paper",
                "Twitter's Manhattan paper"
            ],
            'implement': [
                "A simple Raft implementation",
                "Vector clocks for causal consistency",
                "Power of d choices load balancer",
                "Queueing theory simulator"
            ],
            'experiment': [
                "Measure tail latency in your system",
                "Try different consistency levels",
                "Simulate network partitions",
                "Benchmark lock-free vs lock-based"
            ]
        }

The Big Picture:

Distributed systems theory isn't just academic—it's the foundation of every cloud service, database, and large-scale application. Understanding these concepts helps you:

Make informed design decisions based on trade-offs
Debug complex issues by understanding root causes
Plan for scale using mathematical models
Innovate new solutions by combining ideas

Remember These Principles:

There are no perfect solutions, only trade-offs (CAP, FLP, etc.)
Time and order are fundamental challenges
Scalability requires new ways of thinking
Theory guides, but practice decides

🚀 Your Distributed Systems Journey Continues

You now have a comprehensive understanding of distributed systems theory. But this is just the beginning. The real learning happens when you:

Build systems using these principles
Measure everything and compare to theory
Contribute back to the community

Share your experiences! What theoretical concepts have you applied in practice? Which ones surprised you with their practical relevance? Let's continue the conversation and build better distributed systems together!