Forem: Nasim

Distributed Tracing and W3C Trace Context

Nasim — Mon, 13 Apr 2026 17:45:01 +0000

When a user makes a single request to an application or clicks “checkout” on an e-commerce site, it involves multiple services, API gateway → auth service → cart service → Redis cache → inventory database → payment → notification. Traditional logs show what happened within a single service. Distributed tracing shows the entire story of a request as it traveled across all the services in the application.

The core data model has three concepts:

Trace - journey of request. Each trace has unique trace-id
Span - unit of work within that trace defined with a start time and duration. Could be calling another service, doing a database query. Child spans are also created
Context - the metadata that goes with the request to tell each service that this request is part of another request. This the W3C Trace Context

The Gantt chart (known as a waterfall) immediately shows you the critical path for the application.
Auth → Cart → Inventory → Payment
Reading the Gantt Chart
Each horizontal bar represents one span of work within one service. The left edge of each bar is the start time of that work, the right edge is the finish time, and the width is the length of that span of work. The vertical axis represents the different services, and the horizontal axis is the wall-clock time between 0ms and 600ms.
The vertical lines that are indented within each service represent the parent-child relationships between those services. Each service’s work starts directly when these vertical lines start: it was called directly by its parent service.

Look at the time at when each service starts:

Auth service starts at ~20ms and finishes at ~100ms. API Gateway must authenticate the user before continuing with any other work; Cart cannot start until Auth is complete.
Cart service starts at ~110ms (after Auth) and makes an inline call to Redis (a child service). Cart cannot return successfully to its caller until Redis returns.
Inventory service starts at ~270ms (after Cart) and makes an inline call to Postgres (a child service). Postgres is called inline so that the Inventory service can return to Cart with the requested inventory information.
The Payment service starts at ~410ms (after Inventory is called) and takes 140ms to complete (the longest task after the API Gateway’s root span).
The Notification service starts at ~555ms and is very short at ~25ms in length. This is likely a fire-and-forget service that is called asynchronously when the Purchase operation is complete.

The process strictly follows a sequence: Auth → Cart (with Redis) → Inventory (with Postgres) → Payment → Notification. None of these steps run simultaneously. That’s the main issue—the design enforces a linear chain when, in theory, some parts could operate in parallel.

Now, about why Payment is the bottleneck. A bottleneck isn’t just the slowest step; it’s the part on the critical path that, if improved, would speed up the entire workflow. Here, the longest sequence—the critical path—takes 580ms, and Payment’s 140ms sits toward the end. Cutting time in Payment directly reduces the overall response time. For example, even if you eliminated the 70ms Redis cache retrieval in Cart, Payment would still take 140ms afterward, leaving the total duration almost unchanged.

There’s also another factor: Payment involves calling an external third-party API (Stripe), which adds extra dependency and latency.
You can’t control latency here—it can jump unpredictably. Network round-trips add delays on top of Stripe’s processing time, and any outage or rate limit from Stripe stalls your whole checkout flow.

The amber highlight in the diagram shows it’s slower compared to other parts, but the real problem is where it sits—in a late spot in a chain with no parallel work and relying on an external system you can't tweak.

So, now what? Two main handles: parallelism and externalization.
First, run Inventory and Auth checks at the same time. Starting both simultaneously after the request could cut down about 200ms of sequential waiting to around 130ms, limited by whichever is slower.

Second, take Payment out of the critical path. Accept the order right away, then process payment asynchronously using a queue. This lets users see a quick “order received” response while payment happens behind the scenes—just like most large e-commerce platforms do.

Also, add a timeout and retry strategy. Since Stripe is external, its slowest 1% delays become yours too. Setting a strict timeout—say 200ms—and handling errors gracefully means a Stripe glitch won’t wreck the checkout experience.

Without this trace, you might blame the database or cache, but the waterfall view clears that up instantly.

Now, lets dive into W3C Trace Context

Before W3C settled on a standard in 2021, each had their own way of passing trace context around. Zipkin used X-B3-TraceId headers. AWS X-Ray used X-Amzn-Trace-Id. Jaeger had its own format. When requests moved between these systems, the trace often got lost. W3C Trace Context introduced two HTTP headers that all compliant systems should support

Header 1: traceparent

This is the core carrier. It encodes four fields in a single, compact header:

version — always 00 currently. Future versions can add fields.
trace-id — 128-bit identifier for the entire distributed trace. All spans across all services share this value.
parent-id — 64-bit identifier of the calling span. The receiving service uses this as its parent span when creating its own span.
flags — currently 1 byte. The lowest bit is the sampling flag (01 = sampled, 00 = not sampled).

Header 2: tracestate

Vendor extension list, an ordered comma-separated set of key=value pairs.

Each tracing vendor gets a unique key (e.g. dd for Datadog, b3 for Zipkin B3). This lets vendor-specific metadata (sampling priority, origin, session IDs) ride alongside the standard traceparent without polluting it. When a service forwards a request, it must preserve the full tracestate and may prepend its own entry at the front. Entries beyond 32 are dropped from the tail.

Headers propagation across service boundaries

The sampled flag, which shows as 01 or 00, might seem like a small detail but it actually matters a lot. Tracking every single request in a busy production system is usually too costly—for example, a service handling 100k requests per second. Sampling helps by recording just a slice of traces, yet still keeping full detail for those captured.

The sampled bit (flags 01 vs 00) plays a crucial role in tracing. Tracing every request in a high-traffic system, like one handling 100k requests per second, quickly becomes impractical. Sampling helps by recording only a subset of traces while maintaining detail for those captured.

There are three main sampling methods:
Head-based sampling decides at the root span before downstream calls. It sets the sampled flag to 01 or 00 and propagates it. This approach is straightforward and efficient but can't prioritize capturing interesting events like errors since they haven’t occurred yet.

Tail-based sampling collects all spans from every service. The collector delays the sampling decision until the full trace is received, allowing it to keep all error traces and a small fraction of normal ones. This requires a more complex, stateful collector, such as the OpenTelemetry Collector with tail sampling.

Parent-based sampling simply follows the parent’s sampling decision via the sampled flag. It’s a safe default that ensures the whole trace is either fully recorded or entirely dropped, preventing disconnected spans without a root.

A span isn’t just about its name and how long it lasts. OpenTelemetry, which leads as the open-source implementation of W3C Trace Context, sets semantic conventions—standard attribute names. This helps tracing tools interpret your data consistently, without needing extra setup.

Spans can include events, which are specific log entries tied to a moment within the span, like noting a “cache miss occurred”. They also hold links, which reference other traces—useful in async or queue situations where the producer and consumer traces are related but happen separately.

Async and Queue Propagation
Handling async and queue propagation differs from straightforward HTTP propagation. Since producer and consumer don’t operate simultaneously, you can’t rely on passing a traceparent in a request header.
Instead, the traceparent value is embedded within the message metadata—such as Kafka headers, SQS attributes, or AMQP headers. The consumer then reads this metadata and treats it as a span link, not a parent. This means the consumer’s trace connects to the producer’s but remains a separate trace-id. This approach keeps the causal link clear without letting trace information grow uncontrollably deep.

How does this fit into the OpenTelemetry stack?

As the work-horse OpenTelemetry SDK auto-instruments common frameworks like Express, Django, Spring, and net/http, automatically creating spans and managing traceparent injection and extraction. Each service uses the OTLP exporter to send span data to the OTel Collector, which processes the data—sampling, redacting sensitive info, batching, and distributing it to multiple backends at once. This vendor-neutral setup lets you switch between tools like Jaeger and Grafana Tempo without modifying your application code.

Key implementation considerations

Clock skew — spans from different machines have timestamps from different clocks. A child span with a start time before its parent is a common artifact of NTP drift. Tracing SDKs work around this by using monotonic clocks for durations and accepting some display weirdness on absolute timestamps.
Baggage — a sibling to tracestate, the W3C Baggage header (baggage: userId=alice, env=prod) lets you propagate arbitrary key-value pairs across service boundaries. It's useful for passing tenant IDs or feature flag states without touching application code, but beware: every hop re-serializes and forwards it, so large baggage values add latency.
Instrumentation gaps — if any service in the chain drops the traceparent header (an unmigrated internal proxy, a legacy service, a third-party vendor), the trace breaks. The downstream spans become "orphans" with no parent. Good tracing platforms surface these as disconnected sub-traces.
Security — traceparent is safe to expose externally. tracestate may contain internal vendor metadata you don't want leaking. Strip tracestate at your public ingress, then re-inject a fresh one for internal propagation.
The W3C standard deliberately kept the spec minimal — it defines the wire format, not how to collect, store, or visualize. That separation of concerns is what made it possible for the whole ecosystem to converge on it.

Diagrams created with the assistance of AI.

Ray: Building Distributed Systems—Just Like Running a Restaurant

Nasim — Tue, 14 Oct 2025 16:37:06 +0000

In our fast-paced digital world, data keeps piling up at a staggering rate. To make sense of it all, we need some supercharged computing tools that can handle what we throw at them. That is where Ray comes in. It is an open-source framework designed to help you build distributed applications without all the headaches. In the following sections, we’ll break down each Ray component using our restaurant analogy, translating abstract concepts into familiar, real-world operations. Let’s get cooking!

Ray Data: Ingredient Prep Station
Role: Prepares every ingredient (chopping, washing, stalking) so when the kitchen manager is ready, everything is good to go with the cooking process.
Ray Version: Handles all the preparations for massive data transformations—loading and mapping to transform data sets pre-work to avoid unnecessary time before training.
Key Analogy: If you have a hundred tomatoes to chop or a hundred bags of flour, it is much easier to get it done on Ray Data before it is time to train than during actual cooking.

Ray Train: The Chefs Cooking the Main Dishes
Role: Cooks distribute what they're preparing and they can follow instructions, manage temperature and time, and check in along the way.
Ray Version: Distributes CPU/GPUs for model training, distributes/checks in along the way.
Key Analogy: You want different cooks (hardware) trained on different dishes (models) at the same time as possible so you can kill two birds with one stone and work efficiently.

Ray Tune: The Research Team Experimenting with Recipes
Role: The restaurant's R&D team is constantly exploring new tastes—iterating on sauces, thickness of pasta/ice cream/whatever other courses there are.
Ray Version: Runs many trials in parallel for different parameters to assess best hyperparameters for a model.
Key Analogy: Tune is like a lab that tries dozens or hundreds or iterations on a single recipe to find the best version—most palatable tasting notes—and best version for this request.

Ray Serve: Waitstaff and Quick Service Station
Role: The system for delivering courses to diners in a timely manner at the speed these clients want, especially if there is a rush and volume/inventory can help accommodate.
Ray Version: Deploys and manages models/APIs, auto-scaling for client request routing.
Key Analogy: Serve does not train models because Serve is a waitstaff who does not cook; it takes what is made in the kitchen (Run) to clients and can take many requests at once without sacrificing turnaround time.

Ray RLlib: The Robot Chef Experimenting with Reinforcement Learning
Role: A chef using learning mechanisms over time—finds the best way to create various dishes with notes/feedback along the way.
Ray Version: Builds, distributes/runs reinforcement learning algorithms from agents that can get feedback to learn over time.
Key Analogy: RLlib is a robot chef who can learn through trial and error, figuring out which seasonings people like after adjusting and trying repeatedly.

Ray Core: Kitchen Manager and Scheduling System
Role: Schedules all kitchen staff, knows what each cook is doing, who is available to support what task, ensuring order fulfillment does not bottleneck.
Ray Version: The core component to handle fundamental parallel computing—starting/stopping processes for workers, knowing what's going on, who's got what tasks, and generating reports about assigned resources.
Key Analogy: If one cook is busy, he'll have to wait for the next and it will take longer to line up what's being prepared next but Ray Core will essentially keep everyone in line.

Ray Cluster = The Kitchen: Just like a restaurant has a kitchen where all the cooking happens, a Ray cluster is a collection of computers that work together. They pool their resources, whether it is memory or processing power, to get things done efficiently.

Ray Workers = The Kitchen Staff: In any kitchen, you have distinct roles. There are chefs whipping up meals, sous chefs prepping ingredients, and dishwashers keeping things clean. Ray workers are like that staff—they are computers or components of computers that tackle their tasks independently.

Ray Scheduler = The Head Chef: Think of the head chef as the one in charge of getting things done. They assign tasks to the staff, making sure everyone knows what to work on and when. The Ray scheduler does the same thing by distributing tasks to the workers and keeping everything on track.

Ray Tasks = Recipes: Just like a well-structured recipe guides a chef, Ray tasks are steps that need to be executed. Whether it is crunching data or performing calculations, these tasks are straightforward units of work.

Ray Jobs = Customer Orders: When a customer puts in an order, the kitchen team jumps into action to make it happen. A Ray job is like that order—it is made up of a group of tasks that need to come together to complete a bigger goal, such as training a machine learning model or processing a big dataset.

Ray Actors = Specialized Chefs/Stations: Some chefs have specialties; for instance, a pastry chef only focuses on desserts. In Ray, actors are specialized workers that keep their state between task executions, just like those chefs focus on their craft.

Ray Tasks with Dependencies = Recipe Steps: In cooking, some steps cannot happen until others are finished—like you need to caramelize onions before adding them to the dish. Ray takes care of ordering these tasks properly, ensuring everything gets done at the right time.

Ray Libraries = Specialized Cookbooks: Just as a chef might have cookbooks for various cuisines, Ray comes with built-in libraries that help you deal with common tasks. For instance, it has RLlib for reinforcement learning and Tune for optimizing hyperparameters.

Ray Client = Restaurant Manager/Front of House: Finally, much like a restaurant manager bridges the kitchen and the dining area, the Ray Client makes it easy for users outside the cluster to interact with it.

Ray's Object Store is like a restaurant's central pantry since it streamlines everything all Ray components (tasks/actors/workers) do by facilitating communication and possession of what is needed.

The Central Pantry (Object Store):
The Object Store integrates everything from processed tasks and necessary tasks so there is one place for everyone to come together without issue. In a vast restaurant setting, it is the only way to operate, and a modern-day restaurant must become Ray.
The Ingredients (Objects):
Everything that gets served eventually is in the pantry. It is all there as individual items or groups of items, if necessary, with separate and respective characteristics. In Ray, everything is stored and compartmentalized as an "object".
Access and Sharing:
Since it is a shared pantry, one ingredient can be used by many chefs or dishes and there is no overlap or redundancy. This means that it is easily accessible and reapplied without offense. In Ray, Object Store allows multiple workers to easily access or repurpose what someone else has done so distributed processing is easier and quicker without wasting energy.
Efficiency:
It saves time from searching everywhere and cuts down on ordering redundancy, excess shipping or transporting from other places. Everything they need is in-house. The same with Ray—if the Object Store can pre-exist certain uses, Ray saves on sending over certain pieces of data/transferring network access/computational time of selecting what is used.

Connecting it All — Ray as a Restaurant (single end-to-end flow)

Guest places order (user request / Job starts): someone orders a meal — a job is sent to Ray.
Restaurant manager (Ray Core / Scheduler) parses the order and breaks it down into tasks: Ray Core breaks the job down into tasks and schedules them to workers.
Pantry & prep station (Ray Data) prepare and stage ingredients: data pipelines partition, preprocess, cache, and share datasets for efficient access and effective workers.
Head chef assigns & staff coordinate (Scheduler _+ _Workers / Ray Train): the scheduler gives tasks to the cooks; workers perform distributed compute and training jobs (Ray Train), coordinating efforts for cooking and baking.
Line cooks & sous-chefs execute dishes (Workers / Ray Train): workers perform the compute-heavy steps (model training, batch jobs, data transforms) on the ingredients.
Tasting lab runs experiments (Ray Tune): parallel experiments evaluate various recipe configurations (hyperparameters), feeding better versions back into training.
Robot chef explores improvements continuously (Ray RLlib): reinforcement-learning agents autonomously learn better policy/strategies that improve recipes/models.
Waitstaff & hosts deliver dishes (Ray Serve): Ray Serve hosts the trained model, manages inference requests, scales instances, adapts to traffic — bringing results to the user.
Meal served & feedback loop (Job done → iterate): the order is fulfilled; metrics, feedback fed into Ray Tune, RLlib, retraining, get into action for iterative development.

Ray is like a restaurant that can easily ramp up for a busy night; it provides a means of scaling for computers to work collaboratively with ease on otherwise complicated tasks.

Scaling the Kitchen
Easy Scaling-Ray can add more machines to the cluster, just like a restaurant easily adds in more chefs for a rush, without any issues bringing in one, two, or three more trained professionals to manage a larger situation.
Execution Efficiency-Ray effectively works across many workers creating effective compilation of many hands and minds to cut down on complicated processes—like a kitchen where each team member takes part in a unique, critical role.
Complicated Execution-Ray can manage processes of advanced complexities adding in support to establish an all-encompassing effort—like a restaurant that can put out lots of meals with disparate processes, techniques and timing.

Real-World Examples
Recommendation Systems
Developers implement Ray by working on thousands of computations that require a quick turnaround—from movie recommendation engines that need instantaneous audience analytics to compile the best results.
Scientific Inquiry
Ray engages in many pilot studies, from raw data assessment to model training, in research inquiries seeking simple predictive analytics or extensive exploration into existing data sets.
Conclusion
Ray streamlines distributed computing for developers. When a once-strained restaurant manager can easily expand and organize and optimize service during a rush it creates better opportunities for quality work and better ideas. Similarly, Ray makes scaling and establishing powerful parallel projects to be simple and worthwhile endeavors, transforming developer thinking into reality.

Connect with me on LinkedIn

Is Linux dead or awful

Nasim — Thu, 29 Feb 2024 14:14:04 +0000

The market share of Linux desktops is significantly lower compared to Windows and macOS, which dominate the desktop market. Some technologists have long predicted the downfall of Linux. However, developers, system administrators, and enthusiasts continue to use Linux due to its flexibility, customization options, security features, and the availability of free and open-source software.
Microsoft is still the sultan of OS, their Windows Mobile platform failed to generate interest compared to the pioneering iPhone and Android smartphones, resulting in a lackluster ecosystem with minimal interest. In contrast, Linux is gaining ground in both the mobile and server markets.
Although Linux offers many benefits, such as flexibility, security, and open-source nature, it can be challenging for some users. Different Linux distributions have user interfaces that differ from Windows or macOS, which can be unfamiliar. Software compatibility may require workarounds, and the command-line interface can be daunting for users accustomed to graphical user interfaces. Additionally, while customization is a strength of Linux, it can be overwhelming for new users.
Despite these challenges, the large and active Linux community provides support and documentation. However, finding the right resources and solutions for specific issues can be challenging for beginners. Nonetheless, with the right customization, Linux can be a superior tool if you have nerves, it offers increased privacy, a seamless desktop interface, and enhanced control. The anti-Linux cult has loyal disciples but to be fair all OS's have faults until it fits your use case.