Forem: Nithin Bharadwaj

5 Custom Java Collector Patterns That Replace Complex Stream Reduction Code

Nithin Bharadwaj — Sun, 10 May 2026 09:36:02 +0000

I remember the first time I hit the limits of Java’s Collectors.toList() and Collectors.groupingBy(). I was building a real‑time dashboard for a trading system. The standard collectors could not handle my custom statistics, my multi‑level grouping, or my thread‑safe accumulations. That day I learned to write my own collectors. And it changed the way I look at streams.

Let’s walk through five patterns for writing custom collectors that encapsulate aggregation logic into reusable, composable components. I will keep this as simple as I can, as if I am explaining it to my younger self.

A collector in Java is just a box with four jobs. First, it needs a factory that creates an empty container – that is the supplier. Then it needs a way to drop one element into that container – the accumulator. When streams run in parallel, it needs a method to merge two containers into one – the combiner. And finally, a finisher that transforms the container into the final result you want.

The Java Collector interface captures these four functions plus a set of characteristics that tell the stream how to behave. If you understand these pieces, you can build a collector for almost any aggregation.

I will show you a simple statistics collector that calculates count, sum, minimum, and maximum from a stream of numbers. This pattern is the foundation for everything else.

public class StatisticsCollector<T> implements Collector<T, StatisticsCollector.Accumulator, Statistics> {

    static class Accumulator {
        long count;
        double sum;
        double min = Double.MAX_VALUE;
        double max = Double.MIN_VALUE;
    }

    @Override
    public Supplier<Accumulator> supplier() {
        return Accumulator::new;
    }

    @Override
    public BiConsumer<Accumulator, T> accumulator() {
        return (acc, value) -> {
            double num = ((Number) value).doubleValue();
            acc.count++;
            acc.sum += num;
            acc.min = Math.min(acc.min, num);
            acc.max = Math.max(acc.max, num);
        };
    }

    @Override
    public BinaryOperator<Accumulator> combiner() {
        return (left, right) -> {
            left.count += right.count;
            left.sum += right.sum;
            left.min = Math.min(left.min, right.min);
            left.max = Math.max(left.max, right.max);
            return left;
        };
    }

    @Override
    public Function<Accumulator, Statistics> finisher() {
        return acc -> new Statistics(acc.count, acc.sum, acc.min, acc.max);
    }

    @Override
    public Set<Characteristics> characteristics() {
        return Set.of(Characteristics.CONCURRENT, Characteristics.UNORDERED);
    }

    public static <T extends Number> Collector<T, ?, Statistics> toStatistics() {
        return new StatisticsCollector<>();
    }
}

Notice the CONCURRENT and UNORDERED characteristics. CONCURRENT tells the stream that the accumulator can run safely from multiple threads on the same container – but only if the stream is also unordered. UNORDERED means the order of elements does not matter for the result. Together they allow the stream to skip merging and use a single container for parallel processing. This can be a huge performance win.

But you must be careful. If your accumulator modifies shared state without proper synchronization, you will get corrupted results. In this example, the operations are simple additions and comparisons, which are safe because the accumulator runs on one container at a time (the supplier creates a new container per thread, but with CONCURRENT the supplier might be called only once – check the Java documentation). For truly concurrent collectors, I use atomic classes like LongAdder and AtomicReference. Let me show you that next.

Thread‑Safe Collectors for Parallel Streams

When I write collectors for high‑throughput systems, I use concurrent building blocks. The pattern stays the same, but the accumulator uses thread‑safe objects.

public class ConcurrentStatisticsCollector<T extends Number>
        implements Collector<T, ConcurrentStatisticsCollector.ConcurrentAccumulator, Statistics> {

    static class ConcurrentAccumulator {
        final LongAdder count = new LongAdder();
        final DoubleAdder sum = new DoubleAdder();
        final AtomicReference<Double> min = new AtomicReference<>(Double.MAX_VALUE);
        final AtomicReference<Double> max = new AtomicReference<>(Double.MIN_VALUE);
    }

    @Override
    public Supplier<ConcurrentAccumulator> supplier() {
        return ConcurrentAccumulator::new;
    }

    @Override
    public BiConsumer<ConcurrentAccumulator, T> accumulator() {
        return (acc, value) -> {
            double num = value.doubleValue();
            acc.count.increment();
            acc.sum.add(num);
            acc.min.updateAndGet(current -> Math.min(current, num));
            acc.max.updateAndGet(current -> Math.max(current, num));
        };
    }

    @Override
    public BinaryOperator<ConcurrentAccumulator> combiner() {
        return (left, right) -> {
            left.count.add(right.count.sum());
            left.sum.add(right.sum.sum());
            left.min.updateAndGet(current -> Math.min(current, right.min.get()));
            left.max.updateAndGet(current -> Math.max(current, right.max.get()));
            return left;
        };
    }

    @Override
    public Function<ConcurrentAccumulator, Statistics> finisher() {
        return acc -> new Statistics(
            acc.count.sum(),
            acc.sum.sum(),
            acc.min.get(),
            acc.max.get()
        );
    }

    @Override
    public Set<Characteristics> characteristics() {
        return Set.of(Characteristics.CONCURRENT, Characteristics.UNORDERED);
    }
}

The combiner still merges two accumulators, but it does so by adding the values from both. This is safe because each accumulator was built in its own thread. The CONCURRENT characteristic allows the stream to avoid merging if it knows only one thread will ever call the accumulator – but with atomic fields you are prepared for either case.

I use this pattern when I process millions of events per second. The overhead of atomic operations is small compared to the cost of merging intermediate results.

Multi‑Level Grouping

Grouping by one key is easy. Grouping by two keys and keeping a list of values inside each leaf – that is a custom collector. I needed this when I had to organise sales data by region and then by product category, while preserving the order of individual transactions.

Here is a cascading grouping collector that builds a nested map.

public class CascadingGroupingCollector<T, K1, K2, V>
        implements Collector<T, Map<K1, Map<K2, List<V>>>, Map<K1, Map<K2, List<V>>>> {

    private final Function<? super T, ? extends K1> classifier1;
    private final Function<? super T, ? extends K2> classifier2;

    public CascadingGroupingCollector(
            Function<? super T, ? extends K1> classifier1,
            Function<? super T, ? extends K2> classifier2) {
        this.classifier1 = classifier1;
        this.classifier2 = classifier2;
    }

    @Override
    public Supplier<Map<K1, Map<K2, List<V>>>> supplier() {
        return HashMap::new;
    }

    @Override
    public BiConsumer<Map<K1, Map<K2, List<V>>>, T> accumulator() {
        return (map, item) -> {
            K1 key1 = classifier1.apply(item);
            K2 key2 = classifier2.apply(item);
            map.computeIfAbsent(key1, k -> new HashMap<>())
               .computeIfAbsent(key2, k -> new ArrayList<>())
               .add((V) item);
        };
    }

    @Override
    public BinaryOperator<Map<K1, Map<K2, List<V>>>> combiner() {
        return (left, right) -> {
            right.forEach((key1, innerMap) -> {
                left.merge(key1, innerMap, (existing, incoming) -> {
                    incoming.forEach((key2, list) ->
                        existing.merge(key2, list, (l1, l2) -> {
                            l1.addAll(l2);
                            return l1;
                        }));
                    return existing;
                });
            });
            return left;
        };
    }

    @Override
    public Function<Map<K1, Map<K2, List<V>>>, Map<K1, Map<K2, List<V>>>> finisher() {
        return Function.identity();
    }

    @Override
    public Set<Characteristics> characteristics() {
        return Set.of(Characteristics.UNORDERED);
    }
}

Notice that the finisher returns the map as‑is. That is because the intermediate container is also the final result. This works only if the characteristics do not include IDENTITY_FINISH. If you mark a collector as IDENTITY_FINISH, the stream can skip calling the finisher entirely. Here I deliberately left it out to avoid confusion. In real code, you could add IDENTITY_FINISH and remove the finisher override.

The combiner merges nested maps recursively. It uses merge to handle both levels of keys. This pattern scales to three or more levels – just add another layer of maps and another merge loop.

Top‑N Collector

Sometimes you do not need all the elements – only the ten largest, or the five most recent. The standard toList() gives you everything. A custom top‑N collector keeps only the best items using a bounded TreeSet.

public class TopNCollector<T> implements Collector<T, TreeSet<T>, List<T>> {

    private final Comparator<? super T> comparator;
    private final int n;

    public TopNCollector(Comparator<? super T> comparator, int n) {
        this.comparator = comparator;
        this.n = n;
    }

    @Override
    public Supplier<TreeSet<T>> supplier() {
        return () -> new TreeSet<>(comparator);
    }

    @Override
    public BiConsumer<TreeSet<T>, T> accumulator() {
        return (set, item) -> {
            set.add(item);
            if (set.size() > n) {
                set.pollLast();
            }
        };
    }

    @Override
    public BinaryOperator<TreeSet<T>> combiner() {
        return (left, right) -> {
            for (T item : right) {
                left.add(item);
                if (left.size() > n) {
                    left.pollLast();
                }
            }
            return left;
        };
    }

    @Override
    public Function<TreeSet<T>, List<T>> finisher() {
        return set -> {
            List<T> list = new ArrayList<>(set);
            Collections.reverse(list);
            return list;
        };
    }

    @Override
    public Set<Characteristics> characteristics() {
        return Set.of(Characteristics.UNORDERED);
    }

    public static <T extends Comparable<? super T>> Collector<T, ?, List<T>> topN(int n) {
        return new TopNCollector<>(Comparator.naturalOrder(), n);
    }

    public static <T> Collector<T, ?, List<T>> topN(Comparator<? super T> comparator, int n) {
        return new TopNCollector<>(comparator, n);
    }
}

The magic happens in the accumulator. It adds the element, then immediately removes the worst one if the set exceeds the limit. This keeps memory constant regardless of stream size. The combiner does the same when merging two sets from parallel threads.

I used this collector when I needed to show the top‑10 trending products from a live feed. Instead of storing every product, I kept only the ten with the highest score. The stream never had to hold more than ten plus a few duplicates.

The finisher converts the TreeSet into a list in descending order (because I reversed it). If you want ascending order, skip the reverse.

Composing Collectors with `Teeing`

Java 12 introduced Collectors.teeing to send stream elements to two collectors and merge their results. But its capacity is limited to two. If you need three or four aggregations from a single pass, you can write your own teeing collector that handles any number of branches.

Here is a simplified version that works for two downstream collectors. You can extend it to more by using a list of collectors and a merger that receives a list of results.

public class TeeingCollector<T, R1, R2, R> implements Collector<T, TeeingCollector.Accumulator<R1, R2>, R> {

    static class Accumulator<R1, R2> {
        R1 leftResult;
        R2 rightResult;
    }

    private final Collector<T, ?, R1> left;
    private final Collector<T, ?, R2> right;
    private final BiFunction<? super R1, ? super R2, R> merger;

    public TeeingCollector(
            Collector<T, ?, R1> left,
            Collector<T, ?, R2> right,
            BiFunction<? super R1, ? super R2, R> merger) {
        this.left = left;
        this.right = right;
        this.merger = merger;
    }

    @Override
    @SuppressWarnings("unchecked")
    public Supplier<Accumulator<R1, R2>> supplier() {
        return () -> {
            Accumulator<R1, R2> acc = new Accumulator<>();
            acc.leftResult = ((Supplier<R1>) left.supplier()).get();
            acc.rightResult = ((Supplier<R2>) right.supplier()).get();
            return acc;
        };
    }

    // ... full implementation would include accumulator, combiner, finisher
}

The code above is a skeleton. A complete implementation would store the intermediate containers from each downstream collector. The accumulator would apply both downstream accumulators to each element. The combiner would merge both intermediate containers separately. The finisher would apply the downstream finishers and then the merger.

I built a three‑branch teeing collector for a report that needed counts, sums, and averages from the same stream. Instead of iterating three times, I wrote one collector that ran all three in parallel.

Personal Touches and Practical Advice

When I started writing custom collectors, I made two mistakes. First, I forgot to set the characteristics correctly. If your collector does not have UNORDERED but your stream is ordered, the stream may create more intermediate containers than necessary. Second, I wrote accumulators that were not thread‑safe for parallel streams, leading to rare bugs that only appeared under heavy load.

Test your collectors with both sequential and parallel streams. Use parallelStream() and verify that the result matches. I usually create a small test dataset, then compare the output of a known correct method with the custom collector.

Do not be afraid to inline small collectors with Collector.of(). For quick one‑offs, the factory method is easier than writing a full class. But for reusable logic, writing a dedicated class makes it testable and readable.

Collector<Transaction, ?, Map<Integer, List<Transaction>>> byAmountBucket =
    Collector.of(
        HashMap::new,
        (map, tx) -> map.computeIfAbsent(tx.amount() / 1000, k -> new ArrayList<>()).add(tx),
        (left, right) -> {
            right.forEach((k, v) -> left.merge(k, v, (l1, l2) -> { l1.addAll(l2); return l1; }));
            return left;
        }
    );

This inline collector groups transactions into buckets of thousands. No new class needed.

Summary of the Five Patterns

Statistics Collector – Accumulates multiple numeric metrics in one pass.
Concurrent Collector – Uses atomic classes for safe parallel accumulation.
Cascading Grouping Collector – Builds nested maps with multiple keys.
Top‑N Collector – Keeps only a bounded number of elements using a sorted set.
Teeing Collector – Feeds the stream to multiple downstream collectors and merges.

Each pattern follows the same contract: supplier, accumulator, combiner, finisher. Once you internalise that, you can design collectors for any aggregation problem.

Custom collectors have saved me countless hours of manual iteration and temporary collections. They turn messy loops into clean, declarative pipeline code. And because they implement the Collector interface, they work automatically with parallel streams, giving you free performance gains when your data grows.

The next time you find yourself writing a `for` loop that reduces a stream into a complex structure, stop and ask: can I package this logic into a collector? If the answer is yes, you will end up with code that is easier to read, easier to test, and easier to reuse. I promise you, it is worth the effort.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

8 Web Worker Techniques That Stop JavaScript From Freezing Your Browser

Nithin Bharadwaj — Sun, 10 May 2026 09:21:30 +0000

I remember the exact moment I realized JavaScript had a serious problem. I was building a photo filter app. I applied a blur effect to a large image, and the whole page froze. I couldn't scroll, click, or interact. The browser tab turned white, and the operating system eventually asked me to kill it. That’s when I learned that JavaScript runs on a single thread. Any heavy computation blocks the user interface, making the application feel broken. Web Workers came to my rescue. They let me run scripts in the background without touching the main thread. Over time I discovered eight techniques that turned me from a frustrated developer into someone who could handle massive CPU work without breaking a sweat. I want to share them with you in the simplest way possible.

The first technique is managing the worker lifecycle. A worker is like a tiny program that runs in its own world. You create it, give it a task, and eventually kill it when you are done. But if you just create workers one after another without a plan, you end up with memory leaks and zombie threads. I built a worker factory that makes creating and destroying workers neat. Instead of writing a separate file for every worker, I generate the worker code on the fly using a Blob URL. The factory stores each worker in a map so I can track it. When I finish, I release the memory by revoking the Blob URL and terminating the worker. This keeps my application clean.

class WorkerFactory {
  constructor() {
    this.workers = new Map();
  }

  createFromFunction(fn, options = {}) {
    const code = `self.onmessage = async (e) => {
      const result = await (${fn.toString()})(e.data);
      self.postMessage(result);
    }`;
    const blob = new Blob([code], { type: 'application/javascript' });
    const url = URL.createObjectURL(blob);
    const worker = new Worker(url, options);
    this.workers.set(worker, { url, busy: false });
    return worker;
  }

  createFromURL(url, options = {}) {
    const worker = new Worker(url, options);
    this.workers.set(worker, { url, busy: false });
    return worker;
  }

  terminate(worker) {
    const entry = this.workers.get(worker);
    if (entry) {
      URL.revokeObjectURL(entry.url);
      worker.terminate();
      this.workers.delete(worker);
    }
  }

  terminateAll() {
    for (const [worker, entry] of this.workers) {
      URL.revokeObjectURL(entry.url);
      worker.terminate();
    }
    this.workers.clear();
  }
}

Now that I had control over individual workers, I faced a second problem. If I spawned too many workers at once, the browser would choke. My CPU would scream. So I built a thread pool. A thread pool holds a fixed number of workers, usually matching the number of CPU cores. Tasks go into a queue. Whenever a worker becomes idle, it picks the next task from the queue. This keeps the system busy but not overloaded. I added priority support because some tasks are more urgent than others. The pool sorts the queue by priority before dispatching. This pattern is the backbone of any serious parallel processing in JavaScript.

class WorkerPool {
  constructor(workerFactory, size = navigator.hardwareConcurrency || 4) {
    this.workers = [];
    this.queue = [];
    this.resolvers = new Map();
    this.idCounter = 0;

    for (let i = 0; i < size; i++) {
      const worker = workerFactory.createFromFunction(this.taskRunner);
      worker.onmessage = (e) => this.handleResult(e.data);
      this.workers.push({ worker, busy: false });
    }
  }

  static taskRunner(data) {
    return data;
  }

  execute(data, priority = 0) {
    return new Promise((resolve, reject) => {
      const id = this.idCounter++;
      this.resolvers.set(id, { resolve, reject });
      this.queue.push({ id, data, priority });
      this.dispatch();
    });
  }

  dispatch() {
    if (this.queue.length === 0) return;
    const idleWorker = this.workers.find(w => !w.busy);
    if (!idleWorker) return;

    this.queue.sort((a, b) => b.priority - a.priority);
    const task = this.queue.shift();
    idleWorker.busy = true;
    idleWorker.worker.postMessage({ id: task.id, data: task.data });
  }

  handleResult({ id, result, error }) {
    const resolver = this.resolvers.get(id);
    if (resolver) {
      this.resolvers.delete(id);
      if (error) resolver.reject(new Error(error));
      else resolver.resolve(result);
    }

    for (const w of this.workers) {
      if (w.busy) {
        w.busy = false;
        break;
      }
    }
    this.dispatch();
  }

  terminate() {
    for (const { worker } of this.workers) {
      worker.terminate();
    }
    this.workers = [];
    this.queue = [];
    this.resolvers.clear();
  }
}

The third technique changed my life: transferable objects. When you send data to a worker, JavaScript normally copies it. That copying takes time and memory, especially with large arrays. Transferable objects let you give the data to the worker without copying. The ownership moves from the main thread to the worker. The original array becomes empty instantly. For image processing or binary data, this is a huge speed boost. I remember processing a 4K image in a few milliseconds instead of seconds. You just list the buffers in the second argument of postMessage.

function processImage(imageData) {
  const buffer = imageData.data.buffer;
  const worker = new Worker('image-worker.js');
  worker.postMessage({ imageData: buffer }, [buffer]);
  return new Promise(resolve => {
    worker.onmessage = (e) => {
      const resultBuffer = e.data;
      worker.terminate();
      const resultData = new ImageData(
        new Uint8ClampedArray(resultBuffer),
        imageData.width,
        imageData.height
      );
      resolve(resultData);
    };
  });
}

// image-worker.js
self.onmessage = (e) => {
  const buffer = e.data.imageData;
  const pixels = new Uint8ClampedArray(buffer);
  for (let i = 0; i < pixels.length; i += 4) {
    const gray = 0.3 * pixels[i] + 0.59 * pixels[i+1] + 0.11 * pixels[i+2];
    pixels[i] = gray;
    pixels[i+1] = gray;
    pixels[i+2] = gray;
  }
  self.postMessage(buffer, [buffer]);
};

The fourth technique is SharedArrayBuffer. This was the scary one at first. SharedArrayBuffer lets multiple workers read and write the same memory without sending messages. It’s true parallel processing. But you have to be careful because two workers might write to the same spot at the same time. That causes race conditions. JavaScript gives you atomic operations through the Atomics object to prevent that. I use a simple spinlock to protect critical sections. SharedArrayBuffer is perfect for real-time audio processing or physics simulations where you need constant data flow without the overhead of messages.

const sharedBuffer = new SharedArrayBuffer(1000 * 8);
const sharedArray = new Float64Array(sharedBuffer);
const lock = new Int32Array(sharedBuffer, 0, 1);

function acquireLock() {
  while (Atomics.compareExchange(lock, 0, 0, 1) !== 0) {}
}

function releaseLock() {
  Atomics.store(lock, 0, 0);
}

// Worker 1 writes
acquireLock();
sharedArray[0] = Math.random();
releaseLock();

// Worker 2 reads
acquireLock();
const value = sharedArray[0];
releaseLock();

The fifth technique is task scheduling and prioritization. Sending everything to a pool works, but sometimes you need to pause, cancel, or throttle tasks. I built a scheduler that breaks a big job into small chunks and sends them to the pool one by one. The main thread can check if the browser is idle with requestIdleCallback before scheduling the next chunk. This keeps the UI responsive even during heavy work. If the user navigates away, I cancel all pending tasks. The scheduler gives me fine control over what runs and when.

class Scheduler {
  constructor(pool) {
    this.pool = pool;
    this.pending = new Map();
  }

  schedule(data, priority, onProgress) {
    const id = this.generateId();
    this.pending.set(id, { data, priority, onProgress, cancelled: false });
    this.processQueue();
    return id;
  }

  async processQueue() {
    for (const [id, task] of this.pending) {
      if (task.cancelled) continue;
      this.pending.delete(id);
      try {
        const result = await this.pool.execute(task.data, task.priority);
        if (!task.cancelled && task.onProgress) task.onProgress(1);
        return result;
      } catch (e) {
        console.error(e);
      }
    }
  }

  cancel(id) {
    const task = this.pending.get(id);
    if (task) {
      task.cancelled = true;
      this.pending.delete(id);
    }
  }

  generateId() {
    return Date.now().toString(36) + Math.random().toString(36).substr(2);
  }
}

The sixth technique is error handling and recovery. Workers can crash. Network requests inside a worker can fail. If you don’t handle errors, your whole application might hang. I wrote a function that creates a resilient worker. It listens for the onerror event. When a worker fails, it terminates the broken worker and spins up a new one. It also retries the task up to a maximum number of times. Inside the worker, I wrap the logic in a try-catch and post back an error object. The main thread checks for errors and rejects the promise. This pattern saved me many times during long batch processes.

function createResilientWorker(url, options = {}) {
  const maxRetries = options.maxRetries || 3;
  let retries = 0;

  function startWorker() {
    const worker = new Worker(url);
    worker.onerror = (event) => {
      console.error('Worker error:', event.message);
      if (retries < maxRetries) {
        retries++;
        worker.terminate();
        startWorker();
      }
    };
    worker.onmessageerror = (event) => {
      console.error('Worker message error:', event.data);
    };
    return worker;
  }

  return startWorker();
}

The seventh technique is worker nesting. Sometimes a single worker is not enough. For very large matrix multiplications, I had a master worker that spawned its own child workers. The master receives a big matrix, splits it into chunks, and sends each chunk to a child worker. The children compute in parallel, then the master combines the results. Worker nesting allows hierarchical parallelism. You have to be careful about resource limits, but it works beautifully. I used importScripts inside the master to share helper functions with the children.

// main-worker.js
self.onmessage = function(e) {
  const { matrix, rows, cols } = e.data;
  const chunkSize = Math.floor(rows / navigator.hardwareConcurrency);
  const workers = [];
  const results = [];

  for (let i = 0; i < navigator.hardwareConcurrency; i++) {
    const startRow = i * chunkSize;
    const endRow = (i === navigator.hardwareConcurrency - 1) ? rows : startRow + chunkSize;
    const worker = new Worker('child-worker.js');
    worker.postMessage({ matrix: matrix.slice(startRow * cols, endRow * cols), cols });
    workers.push(worker);
  }

  workers.forEach((worker, index) => {
    worker.onmessage = (e) => {
      results[index] = e.data;
      if (results.length === workers.length) {
        const combined = results.flat();
        self.postMessage(combined);
        workers.forEach(w => w.terminate());
      }
    };
  });
};

The eighth technique is progress reporting. Users hate staring at a blank screen. They want to know how far along a task is. Inside a worker, I send periodic messages with a progress value between 0 and 1. The main thread updates a progress bar. I throttle the messages so they don’t flood the main thread. For example, I send a progress update only every 100 iterations. This keeps the UI responsive and gives the user a sense of progress. I even send an estimated time remaining. It makes the application feel professional and trustworthy.

// worker.js
self.onmessage = async (e) => {
  const total = e.data.total;
  for (let i = 0; i < total; i++) {
    await new Promise(resolve => setTimeout(resolve, 10));
    if (i % 100 === 0) {
      self.postMessage({ progress: i / total });
    }
  }
  self.postMessage({ progress: 1, done: true });
};

// main.js
const worker = new Worker('worker.js');
worker.onmessage = (e) => {
  if (e.data.progress !== undefined) {
    updateProgressBar(e.data.progress);
    if (e.data.done) {
      console.log('Complete');
    }
  }
};

These eight techniques gave me the power to write applications that handle millions of data points without freezing. But the story does not end there. I also needed to integrate this with React. I created a custom hook that wraps a worker pool. The hook creates the pool when the component mounts and destroys it on unmount. It exposes an execute function and a status state. Inside the hook, I handle loading, success, and error states gracefully. Now I can use Web Workers in any React component with just a few lines.

function useWorkerPool(workerFactory, poolSize) {
  const poolRef = useRef(null);
  const [status, setStatus] = useState('idle');

  useEffect(() => {
    poolRef.current = new WorkerPool(workerFactory, poolSize);
    return () => poolRef.current?.terminate();
  }, [workerFactory, poolSize]);

  const execute = useCallback(async (data) => {
    setStatus('running');
    try {
      const result = await poolRef.current.execute(data);
      setStatus('idle');
      return result;
    } catch (error) {
      setStatus('error');
      throw error;
    }
  }, []);

  return { execute, status };
}

I use this hook in my image processing app. The main thread never blocks. The user can scroll and click while a heavy filter applies. The progress bar fills smoothly. I even combined SharedArrayBuffer with a thread pool to build a real-time audio visualizer. The experience changed how I see JavaScript. It is not just a language for clicking buttons. It can do serious number crunching. But you have to treat the workers with respect. Manage their lifecycles. Pool them. Transfer data efficiently. Protect shared memory. Handle errors. Report progress. When you do all that, you get a responsive, fast, and professional application.

I hope you try these techniques yourself. Start with a simple worker factory and a thread pool. Then experiment with transferable objects. Once you feel comfortable, explore SharedArrayBuffer. Build a scheduler. Make your workers resilient. Try nesting them. Add a progress bar. And finally, integrate it all into your favorite framework. Your users will thank you. Your CPU will run cooler. And you will never fear a long computation again.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Build Custom Chrome DevTools Extensions for Real-Time Debugging

Nithin Bharadwaj — Sun, 10 May 2026 09:11:13 +0000

I remember the first time I tried to build something that lived inside Chrome DevTools. I had this weird idea—a panel that would show me every time a variable changed in my React app. I knew content scripts could touch the page, but I wanted the panel to feel native, like the Elements tab or the Console. So I started digging into the extension APIs, and I found out that the browser opens a whole set of tools just for people like us, who want to make custom debugging gadgets. Let me show you what I learned.

The whole thing begins with a plain JSON file. Your extension needs a manifest, and inside it you declare a "devtools_page". This is just an HTML file that will be loaded when the user opens DevTools. But that page’s real job is to register your custom panels. So you create a small HTML file like devtools.html that doesn’t have much content itself—it just calls chrome.devtools.panels.create. That API takes a name, an icon, and the path to another HTML file that will display your panel’s UI. The permissions are simple: you need "storage" if you want to save settings, "tabs" to communicate with the inspected page, and "activeTab" for one-off access. You don’t need the broad "devtools" permission; the "devtools_page" field is enough.

{
  "name": "My Custom Inspector",
  "version": "1.0.0",
  "devtools_page": "devtools.html",
  "permissions": ["storage", "tabs", "activeTab"],
  "background": { "scripts": ["background.js"] }
}

In devtools.html you might write just a script that creates the panel:

chrome.devtools.panels.create(
  "My Panel",
  "icon.png",
  "panel.html"
);

Now when you open DevTools, you’ll see a new tab named "My Panel". That’s where your custom interface lives. But getting data from the webpage into that panel requires a little dance.

The inspected page is separate from your DevTools extension. You can’t just throw a script tag into it from the panel. Instead, you use chrome.devtools.inspectedWindow.eval(). This function runs JavaScript code inside the page’s context and returns the result. I use this to grab any global state I care about—like a Redux store snapshot, or a list of all registered service workers. The second parameter is an options object where you can set useContentScriptContext: true if you want the code to run with the same permissions as a content script. That’s helpful for accessing DOM elements that content scripts can see.

function getAppState() {
  chrome.devtools.inspectedWindow.eval(
    `JSON.stringify(window.__APP_STATE__)`,
    { useContentScriptContext: true },
    (result, isException) => {
      if (!isException) {
        updatePanel(JSON.parse(result));
      }
    }
  );
}

But eval only gives you a one-shot communication. If you want the panel to receive real-time updates from the page—like when a user clicks a button—you need a permanent bridge. That’s where the background script comes in. Open a port from the panel to the background using chrome.runtime.connect. Then in the background script, forward messages to a content script that lives in the inspected tab. The content script can listen for events on the page and send them back. It’s a round trip, but it works reliably.

// panel.js
const port = chrome.runtime.connect({ name: 'panel-background' });
port.onMessage.addListener(msg => displayEvent(msg));

// background.js
chrome.runtime.onConnect.addListener(port => {
  port.onMessage.addListener((msg, sender) => {
    chrome.tabs.sendMessage(sender.sender.tabId, msg, response => {
      port.postMessage(response);
    });
  });
});

That bridge becomes the backbone for custom network inspectors. The DevTools extension API gives you chrome.devtools.network which can listen for every HTTP request that happens in the inspected page. I use onRequestFinished to catch the response details. The callback gives you a request object with methods to get the response body. I filter for API calls and log them in my panel. But you can go further. There’s also onBeforeRequest which fires before the request is sent. You can modify request headers there. That’s handy if you want to add a debug header to every API call, or block certain resources during testing.

// Listen for completed requests
chrome.devtools.network.onRequestFinished.addListener(request => {
  if (request.request.url.includes('/api/')) {
    request.getContent(content => {
      logApiCall({
        url: request.request.url,
        method: request.request.method,
        status: request.response.status,
        body: content
      });
    });
  }
});

// Modify headers before request is sent
chrome.devtools.network.onBeforeRequest.addListener(details => {
  details.requestHeaders.push({ name: 'X-Debug', value: 'true' });
  return { requestHeaders: details.requestHeaders };
});

Performance profiling inside a custom panel is surprisingly easy. The page can use the standard performance.mark() and performance.measure() methods, and your panel can read them using inspectedWindow.eval. I once built a tool that measured how long each step in a file upload process took. I marked the start and end in the page, then on the panel side I pulled all the measures and rendered a simple horizontal bar chart. No need for the complex performance timeline API; the user’s own markers give you exactly what you need.

// In the page script (injected or existing)
performance.mark('upload-start');
fetch('/upload', { method: 'POST', body: file })
  .then(() => performance.mark('upload-end'))
  .then(() => performance.measure('upload-time', 'upload-start', 'upload-end'));

// In panel.js
chrome.devtools.inspectedWindow.eval(
  `JSON.stringify(performance.getEntriesByType('measure'))`,
  result => renderTimeline(JSON.parse(result))
);

DOM inspection panels can go way beyond what the default Elements tab shows. You can query any element in the page, get its computed styles, bounding rectangle, event listeners, even accessibility properties. I built a panel that highlights elements with a specific CSS class and shows their z-index stacking context. The trick is to use inspectedWindow.eval to run a function that returns a plain object with only the data you need. You can’t send DOM nodes directly because they’re not serializable. So you extract the relevant properties.

function inspectElement(selector) {
  chrome.devtools.inspectedWindow.eval(
    `(function() {
      const el = document.querySelector('${selector}');
      if (!el) return null;
      const rect = el.getBoundingClientRect();
      const styles = window.getComputedStyle(el);
      return {
        tag: el.tagName,
        classes: Array.from(el.classList),
        id: el.id,
        rect: [rect.left, rect.top, rect.width, rect.height],
        color: styles.color,
        fontSize: styles.fontSize,
        zIndex: styles.zIndex
      };
    })()`,
    result => {
      if (result) highlightInPage(result);
    }
  );
}

Sidebar panes are a special kind of panel that lives inside the Elements tab. When you select a DOM node in the Elements panel, a sidebar can show you extra information about that node. I use chrome.devtools.panels.elements.createSidebarPane and then listen for onSelectionChanged. Inside that listener, I evaluate an expression that uses $0—a special DevTools variable that holds the currently selected element. Then I call sidebar.setObject() to display the result as a JSON tree. This is perfect for showing computed data attributes, event listeners, or any custom metadata attached to the element.

chrome.devtools.panels.elements.createSidebarPane("Custom Props", sidebar => {
  chrome.devtools.panels.elements.onSelectionChanged.addListener(() => {
    chrome.devtools.inspectedWindow.eval(
      `(function() {
        const el = $0;
        if (!el) return null;
        return {
          dataset: Object.assign({}, el.dataset),
          attributes: Array.from(el.attributes).map(a => ({name: a.name, value: a.value}))
        };
      })()`,
      result => sidebar.setObject(result || { error: 'No element selected' })
    );
  });
});

Sometimes the extension APIs aren’t enough. For example, if you want to enable raw network traffic capture or manipulate the DOM tree in ways that the API doesn’t allow, you can connect directly to Chrome DevTools Protocol (CDP) via a WebSocket. You need to know the WebSocket debug URL of the inspected page. You can get that from chrome.debugger API or by starting a remote debugging session. Once you have the URL, you open a WebSocket and send JSON commands like "Network.enable" or "DOM.getDocument". This gives you the same power as the full DevTools frontend. I used this to build a custom profiler that recorded every HTTP request with precise timestamps that the high-level API didn’t expose.

// background.js
const ws = new WebSocket('ws://localhost:9222/devtools/page/PAGE_ID');
ws.onopen = () => {
  ws.send(JSON.stringify({ id: 1, method: 'Network.enable' }));
  ws.send(JSON.stringify({ id: 2, method: 'DOM.getDocument' }));
};
ws.onmessage = event => {
  const msg = JSON.parse(event.data);
  if (msg.method === 'Network.requestWillBeSent') {
    console.log(msg.params.request.url);
  }
};

Finally, a real tool has to feel like part of DevTools, not a foreign popup. Save user preferences to chrome.storage.local so settings persist between sessions. I always store the last active filter, the panel size, and any logs the user chose to keep. Also, listen for theme changes with chrome.devtools.panels.onThemeChanged. When the user switches DevTools from light to dark mode, your panel’s CSS variables should update automatically. I set a data attribute on the body and let CSS handle the rest.

// Save state
chrome.storage.local.set({ filter: 'error', collapsed: true });

// Load state
chrome.storage.local.get(['filter', 'collapsed'], items => {
  applyFilter(items.filter);
  setCollapsed(items.collapsed);
});

// Theme adaptation
chrome.devtools.panels.onThemeChanged.addListener(theme => {
  document.body.dataset.theme = theme; // 'default' or 'dark'
});

Building these panels taught me that the browser gives you a surprising amount of freedom. You are not limited to just injecting content scripts. You can create full-blown debugging environments that integrate with every part of DevTools. The manifest sets the stage, the inspectedWindow API gives you access to the page, the network hook lets you see traffic, and the CDP open the door to the very core of the browser. With message passing, you can build real-time dashboards. With sidebars, you can inspect elements in ways that even the default tools don’t offer. And with persistence and theming, your tool feels like a first-class citizen.

I encourage you to start small. Create a panel that prints the current URL. Then expand it to show something specific to your project, like the number of open WebSocket connections or the current route. Once you taste that power, you will never look at DevTools the same way again. It’s not just a tool—it’s a platform you can extend. And now you have the techniques to do it.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Python Metaprogramming: 8 Powerful Techniques That Power Django, SQLAlchemy, and Pydantic

Nithin Bharadwaj — Sat, 09 May 2026 10:52:35 +0000

Let me tell you something about Python that took me years to fully appreciate. You can write code that writes code. Not in some abstract, philosophical sense, but literally. You can create classes while your program runs. You can intercept every attribute access and decide what happens. You can build functions from strings generated from configuration files. This is metaprogramming, and it is the superpower that makes frameworks like Django ORM, SQLAlchemy, and Pydantic work. I will walk you through eight techniques, with code you can run and modify. I learned these by breaking things and fixing them, so I will share the mistakes I made too.

Let me start with metaclasses. A metaclass is the class of a class. Just like an instance is created from a class, a class is created from a metaclass. By default, Python uses type as the metaclass. You can create your own metaclass and inject behavior every time a new class is defined.

I remember the first time I needed this. I had ten model classes, and I kept forgetting to add a created_at timestamp. I could have copied the same line into every class, but that is boring and error prone. Instead, I wrote a metaclass that automatically adds the timestamp attribute.

import time

class TimestampMeta(type):
    def __new__(mcs, name, bases, namespace):
        namespace['_created_at'] = time.time()
        return super().__new__(mcs, name, bases, namespace)

class BaseModel(metaclass=TimestampMeta):
    pass

class User(BaseModel):
    def __init__(self, name):
        self.name = name

u = User("Alice")
print(u._created_at)  # 1640000000.123

Notice that _created_at is set when the class is defined, not when an instance is created. That means all instances of User share the same timestamp? Yes, because it is a class attribute. If you need per-instance timestamps, you need to hook into __init__ instead. But the metaclass runs before any instance exists, so you can also inject methods or modify existing ones.

Let me modify the previous example to rename all methods automatically:

class RenameMeta(type):
    def __new__(mcs, name, bases, namespace):
        new_namespace = {}
        for key, value in namespace.items():
            if callable(value) and not key.startswith('__'):
                new_namespace['auto_' + key] = value
            else:
                new_namespace[key] = value
        return super().__new__(mcs, name, bases, new_namespace)

class Service(metaclass=RenameMeta):
    def run(self):
        return "running"

s = Service()
print(s.auto_run())  # "running"

This is useful when you build plugins or DSLs that require all public methods to follow a naming convention. You can also add validation: raise an error if a class does not implement a required method.

Now, descriptors. Descriptors control how attributes are accessed and set on an instance. Properties are built-in descriptors, but you can create your own. The magic methods are __get__, __set__, and __delete__.

I once needed to ensure that certain attributes were always positive integers. I could have written a setter in every class, but that duplicates logic. Instead, I created a descriptor that validates on assignment.

class PositiveInteger:
    def __init__(self):
        self.data = {}

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return self.data.get(id(obj), None)

    def __set__(self, obj, value):
        if not isinstance(value, int) or value <= 0:
            raise ValueError(f"{value} is not a positive integer")
        self.data[id(obj)] = value

    def __delete__(self, obj):
        del self.data[id(obj)]

class ShoppingCart:
    quantity = PositiveInteger()
    price = PositiveInteger()

    def __init__(self, quantity, price):
        self.quantity = quantity
        self.price = price

cart = ShoppingCart(3, 1500)
cart.quantity = 5  # OK
# cart.quantity = -1  # ValueError

You may wonder why I stored values in a dictionary keyed by id(obj). Because the descriptor is a class attribute shared by all instances. If I stored value directly, every instance would overwrite the same storage. Using id(obj) gives each instance its own slot. This is exactly how properties work under the hood.

I used this pattern later to build lazy-loaded attributes that compute once and cache the result.

class LazyProperty:
    def __init__(self, func):
        self.func = func
        self.name = func.__name__
        self.cache = {}

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if id(obj) not in self.cache:
            self.cache[id(obj)] = self.func(obj)
        return self.cache[id(obj)]

class Report:
    def __init__(self, data):
        self.data = data

    @LazyProperty
    def summary(self):
        print("Computing summary (expensive)...")
        return sum(self.data) * 2

r = Report([1, 2, 3])
print(r.summary)  # computes and prints "Computing...", returns 12
print(r.summary)  # returns 12, no print

Now, __getattr__ and __setattr__. These hooks allow you to intercept attribute access on instances. __getattr__ is called only when the attribute is not found through normal means. __getattribute__ is called for every attribute access, including existing ones. Use __getattr__ for dynamic proxies.

I used this to create a wrapper around a remote API. The wrapper does not know the method names in advance; it sends whatever method you call as a JSON request.

class RemoteAPI:
    def __init__(self, base_url):
        self.base_url = base_url

    def __getattr__(self, name):
        def method(*args, **kwargs):
            print(f"Sending request to {self.base_url}/{name}")
            # Simulate a real request
            return {"method": name, "args": args, "kwargs": kwargs}
        return method

api = RemoteAPI("https://api.example.com")
result = api.getUser(id=42)
print(result)

But be careful with recursion. If __getattr__ tries to access self.something that does not exist, it will call __getattr__ again infinitely. Always use object.__getattribute__ for default behavior.

class SafeProxy:
    def __getattr__(self, name):
        if name.startswith('_'):
            raise AttributeError(name)
        # fallback to default
        return object.__getattribute__(self, name)

Now, dynamic code generation with exec and compile. This is the most dangerous but also the most powerful technique. You can build functions from strings defined in configuration files or databases.

I wrote a validation system once where the rules came from a YAML file. I needed to generate functions that check constraints without hardcoding them.

def make_validator(rules):
    code_lines = ["def validator(value):"]
    for rule in rules:
        if rule['type'] == 'min_length':
            code_lines.append(f"    if len(value) < {rule['value']}:")
            code_lines.append(f"        raise ValueError('Too short')")
        if rule['type'] == 'regex':
            code_lines.append(f"    import re")
            code_lines.append(f"    if not re.match(r'{rule['pattern']}', value):")
            code_lines.append(f"        raise ValueError('Invalid format')")
    code_lines.append("    return value")
    code = "\n".join(code_lines)
    namespace = {}
    exec(code, namespace)
    return namespace['validator']

rules = [
    {'type': 'min_length', 'value': 3},
    {'type': 'regex', 'pattern': '^[a-zA-Z]+$'},
]
validator = make_validator(rules)
print(validator("abc"))  # "abc"
# validator("ab")  # raises ValueError

Always limit what exec can access. Supply an empty dictionary as the namespace to prevent the generated code from modifying your real variables. If you need to allow imports, whitelist them carefully.

The inspect module lets you examine functions and classes at runtime. You can extract argument names, defaults, annotations, and even source code. I used it to build an automatic input sanitizer that reads type hints.

import inspect

def typed(func):
    sig = inspect.signature(func)
    hints = func.__annotations__

    def wrapper(*args, **kwargs):
        bound = sig.bind(*args, **kwargs)
        bound.apply_defaults()
        for name, value in bound.arguments.items():
            if name in hints:
                expected = hints[name]
                if not isinstance(value, expected):
                    raise TypeError(f"{name} must be {expected.__name__}, got {type(value).__name__}")
        result = func(*args, **kwargs)
        if 'return' in hints:
            expected = hints['return']
            if not isinstance(result, expected):
                raise TypeError(f"Return must be {expected.__name__}, got {type(result).__name__}")
        return result
    return wrapper

@typed
def divide(a: int, b: int) -> float:
    return a / b

print(divide(10, 3))  # 3.3333
# divide("10", 3)  # TypeError

This is similar to how modern libraries enforce types, but with less overhead. You can extend it to coerce values, log calls, or cache results.

Now, dynamic class creation with type. Instead of writing class definitions in your source code, you can call type(name, bases, dict) to create a class at runtime. This is how ORMs like SQLAlchemy create models from table definitions.

I built a small configuration-driven data model system. The user provides a list of fields, and I generate a class with those fields, a constructor, and a readable representation.

def create_model(name, fields):
    annotations = {}
    init_lines = ["def __init__(self, {}):".format(
        ", ".join(f"{fname}: {ftype.__name__ if hasattr(ftype, '__name__') else ftype}" for fname, ftype, _ in fields)
    )]
    for fname, _, default in fields:
        if default is not None:
            init_lines.append(f"    self.{fname} = {fname}")
        else:
            init_lines.append(f"    self.{fname} = {fname}")
        annotations[fname] = _
    init_code = "\n".join(init_lines)
    namespace = {}
    exec(init_code, namespace)

    def __repr__(self):
        return f"{name}({', '.join(f'{k}={v!r}' for k,v in self.__dict__.items())})"

    cls = type(name, (object,), {
        '__annotations__': annotations,
        '__init__': namespace['__init__'],
        '__repr__': __repr__,
    })
    return cls

Person = create_model("Person", [
    ("name", str, None),
    ("age", int, None),
    ("active", bool, True)
])

p = Person(name="Bob", age=30, active=True)
print(p)  # Person(name='Bob', age=30, active=True)

The exec inside create_model is safe because the namespace is empty and we control the code string completely. But you must be careful with user-supplied field names that could contain malicious strings.

Now, the ast module. Parsing Python source into an Abstract Syntax Tree lets you analyze and transform code before it runs. You can build static analyzers, linters, or simple optimizers.

I once wanted to precompute constant expressions in a large config file to speed up startup. I used ast.NodeTransformer to fold arithmetic expressions.

import ast

class Folder(ast.NodeTransformer):
    def visit_BinOp(self, node):
        self.generic_visit(node)
        if isinstance(node.left, ast.Constant) and isinstance(node.right, ast.Constant):
            if isinstance(node.op, ast.Add):
                return ast.Constant(node.left.value + node.right.value)
            elif isinstance(node.op, ast.Mult):
                return ast.Constant(node.left.value * node.right.value)
        return node

def compile_with_folding(source):
    tree = ast.parse(source)
    folded = Folder().visit(tree)
    ast.fix_missing_locations(folded)
    code = compile(folded, '<string>', 'exec')
    ns = {}
    exec(code, ns)
    return ns

ns = compile_with_folding("x = 2 + 3 * 4")
print(ns['x'])  # 14, computed at compile time

You can see the original 2 + 3 * 4 becomes 14 before execution. The ast module is also used by tools like Black and Pylint to understand your code without running it.

Finally, context managers and contextvars. Sometimes you need to pass a global-like state across function calls without using globals. contextvars is like a thread-local variable but working with async code too. Combined with contextmanager, you can create scoped resources.

I used this for a logging system where each request has a unique ID. Instead of passing the ID through every function, I set it in a context variable at the beginning of the request.

from contextvars import ContextVar
from contextlib import contextmanager

request_id = ContextVar('request_id', default=None)

@contextmanager
def request_context(rid):
    token = request_id.set(rid)
    try:
        yield
    finally:
        request_id.reset(token)

def log(message):
    rid = request_id.get()
    if rid:
        print(f"[{rid}] {message}")
    else:
        print(message)

with request_context("abc123"):
    log("Processing payment")  # [abc123] Processing payment

log("Outside context")  # Outside context

Notice I did not use yield from or complicated setup. The contextmanager decorator turns a generator function into a context manager. The ContextVar provides a clean way to store per-context data.

These eight techniques have saved me from writing thousands of lines of repetitive code. Each one solves a different problem. Metaclasses for class creation hooks. Descriptors for attribute access control. __getattr__ for dynamic proxies. exec for generating functions from config. inspect for runtime introspection. type for dynamic classes. ast for code analysis. Context managers and contextvars for scoped state.

Start small. Pick one technique and try it in a side project. Add a metaclass that logs every class creation. Or create a descriptor that caps string length. You will see how Python bends to your will without sacrificing readability.

I once broke my entire test suite by misusing __getattr__ (infinite recursion, as I mentioned). Now I always add a guard. I also learned to never use exec on user input without sandboxing. But within controlled boundaries, these tools are safe and liberating.

If you write a framework, a library, or even a complex application, you will eventually need to generate code dynamically or intercept behavior you did not plan for. These patterns are the foundation of Python's flexibility. They are not magic tricks; they are deliberate design choices built into the language.

I hope you take the time to experiment with each one. Write a script that generates an entire ORM from a schema. Or build a proxy that mocks a web service. The code examples I gave are simple enough to run, but they are starting points. Modify them, break them, fix them. That is how you learn.

Remember, the goal is not to use metaprogramming everywhere. The goal is to know when it is the cleanest solution. For most code, plain functions and classes are enough. But when you face a problem that demands flexibility at runtime, these techniques will be your tools.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

# Build Real-Time Data Pipelines With Python, Faust, and Apache Kafka: 8 Proven Techniques

Nithin Bharadwaj — Sat, 09 May 2026 10:38:47 +0000

I remember the first time I tried to handle live data. A stream of clicks from our website was piling up in a log file, and by the time I processed it, the day was over. Real-time streaming changed everything. Instead of waiting, you act the moment an event happens. In Python, Faust and Apache Kafka give you the power to build these pipelines. I’ll walk you through eight techniques that turned my messy data into a fast, reliable stream processor. Each technique is a tool you can pick up and use right now.

1. Defining Stream Processors with Faust

The heart of your pipeline is an agent – a function that runs forever, listening to a Kafka topic. Every message triggers it. Faust calls these agents, and they run as async functions inside a worker app. You tell the app which Kafka broker to talk to, how to serialize your data, and where to keep local state.

import faust

app = faust.App(
    'streaming-pipeline',
    broker='kafka://localhost:9092',
    value_serializer='json',
    store='rocksdb://',
)

class ClickEvent(faust.Record):
    user_id: str
    page: str
    timestamp: float

click_topic = app.topic('clicks', value_type=ClickEvent)

@app.agent(click_topic)
async def process_clicks(clicks):
    async for click in clicks:
        # Process each click event
        print(f"User {click.user_id} visited {click.page}")
        # Enrich and forward to another topic
        enriched = {'user': click.user_id, 'url': click.page}
        await enrich_topic.send(value=enriched)

# Run: faust -A main worker -l info

I used to think I needed a separate framework for every minor change. With Faust, the agent loops over each incoming message. You write normal Python code inside the loop. The worker automatically commits offsets after each successful iteration, so if the process crashes, it restarts from where it left off. The async nature means one worker can handle thousands of messages per second without blocking.

2. Stateful Stream Processing

Sometimes you need to remember things. How many times has this user clicked? What was the last page they saw? Faust gives you a Table – a key-value store backed by RocksDB. The table lives on disk, so even if your worker restarts, the counts survive.

# Count page views per user
page_views = app.Table('page_views', default=int)

@app.agent(click_topic)
async def count_views(clicks):
    async for click in clicks:
        page_views[click.user_id] += 1
        current = page_views[click.user_id]
        if current % 100 == 0:
            print(f"User {click.user_id} has {current} views")

I added a table to my click pipeline to track users hitting 100, 200 views – it felt like magic. You can share tables across agents. A second agent can join clicks with user profiles stored in another table:

user_profiles = app.Table('user_profiles', default=dict, partitions=8)

@app.agent(profile_topic)
async def update_profiles(profiles):
    async for profile in profiles:
        user_profiles[profile['user_id']] = profile

@app.agent(click_topic)
async def join_clicks(clicks):
    async for click in clicks:
        profile = user_profiles.get(click.user_id, {})
        # Combine click with profile data
        print(f"{profile.get('name')} clicked {click.page}")

The table partitions data so it stays close to the agent processing that partition. This keeps joins fast.

3. Windowed Aggregations

Counting total clicks is fine, but often you need time windows – how many clicks per minute? Per hour? Faust supports tumbling windows (non-overlapping) and hopping windows (overlapping).

from datetime import timedelta

# Tumbling window (non-overlapping)
windowed_views = app.Table(
    'windowed_views',
    default=int,
    window=app.tumbling(timedelta(minutes=1)),
)

@app.agent(click_topic)
async def window_count(clicks):
    async for click in clicks:
        windowed_views[click.page] += 1

I used a hopping window for rolling averages on sensor data. Each window covers five minutes but slides every minute – you get smooth, overlapping snapshots.

rolling_avg = app.Table(
    'rolling_avg',
    default=float,
    window=app.hopping(timedelta(minutes=5), timedelta(minutes=1)),
)

@app.agent(metric_topic)
async def compute_rolling(metrics):
    async for m in metrics:
        current = rolling_avg[m['sensor']]
        print(f"Sensor {m['sensor']} window value: {current}")

Internally, Faust cleans up old windows automatically. You set the retention, and it handles the rest.

4. Stream-Stream Joins

Sometimes you need to match two event streams. For example, when an order arrives, you want to check if a confirmation comes within an hour. Faust lets you join two topics using a table or a time-based join.

orders_topic = app.topic('orders', value_type=Order)
confirmations_topic = app.topic('confirmations', value_type=Confirmation)

# Using a table to store confirmations
confirmations_table = app.Table('confirmations', default=dict, window=app.tumbling(timedelta(hours=1)))

@app.agent(confirmations_topic)
async def store_confirmations(confs):
    async for conf in confs:
        confirmations_table[conf.order_id] = conf

@app.agent(orders_topic)
async def match_orders(orders):
    async for order in orders:
        conf = confirmations_table.get(order.order_id)
        if conf:
            print(f"Order {order.order_id} confirmed at {conf.timestamp}")

The window on the confirmation table means old confirmations expire. You can also use the built-in join operator that matches within a time range:

@app.agent(orders_topic)
async def join_within_window(orders):
    await orders.join(
        confirmations_topic,
        within=timedelta(hours=1),
        on_match=lambda order, conf: print(f"Matched {order.id}"),
        on_mismatch=lambda order: print(f"Unmatched {order.id}"),
    )

I use this pattern to detect abandoned carts – orders without a matching payment within ten minutes.

5. Error Handling and Dead Letter Queues

Bad data happens. A missing field, a malformed JSON, a temporary database hiccup. If your agent crashes, the entire pipeline can stall. Faust gives you two strategies: continue or stop.

@app.agent(click_topic, on_error='continue')
async def resilient_processing(clicks):
    async for click in clicks:
        try:
            if click.page is None:
                raise ValueError("Missing page")
            print(f"OK: {click.user_id}")
        except Exception as e:
            # Send to dead letter topic
            await dead_letter_topic.send(
                value={'error': str(e), 'data': click}
            )

I set on_error to 'continue' for most agents. The worker skips the bad message and logs it. To avoid losing data, I route those messages to a separate Kafka topic (dead letter queue). Later, I can reprocess them after fixing the issue.

For agents where data integrity matters more than throughput, you can set on_error='stop'. The worker pauses until you fix the problem manually. Combine it with retries:

@app.agent(click_topic, on_error='stop')
async def strict_processing(clicks):
    async for click in clicks:
        await process_with_retries(click)

async def process_with_retries(click, max_retries=3):
    for attempt in range(max_retries):
        try:
            return await risky_operation(click)
        except:
            await asyncio.sleep(0.5 * attempt)
    raise

The dead letter queue and retry patterns saved me countless hours of debugging.

6. Watermarks and Event Time Processing

Messages often arrive late. A user’s click might be timestamped five minutes ago but reaches the broker only now. If you process based on processing time, you count that click in the current window, not the window it belongs to. Faust supports event time: you tell it which field holds the original timestamp, and it uses watermarks to handle lateness.

@app.agent(click_topic, processing_time=0.1)
async def event_time_aggregation(clicks):
    async for click in clicks:
        watermarked = app.Table(
            'watermarked',
            default=int,
            window=app.tumbling(timedelta(minutes=5)),
            expire=timedelta(minutes=10),  # allow 10 min lateness
        )
        watermarked[click.page] += 1

You set the expire parameter to how much late data you want to accept. The watermark advances as you see timestamps, and windows close only after the watermark passes their end. I use this for IoT sensors that sometimes buffer data offline. Without event time, those delayed readings would distort the hourly average.

7. Partition Rebalancing and Scaling

As your traffic grows, you add more worker instances. Kafka partitions get redistributed among workers. Faust handles this automatically, but you might need custom logic for hot keys. You can write your own assignor.

class CustomAssignor(faust.assignors.Assignor):
    def assign(self, cluster):
        workers = list(cluster.workers)
        partitions = list(cluster.partitions)
        assignments = {}
        for i, p in enumerate(partitions):
            worker = workers[i % len(workers)]
            assignments.setdefault(worker, []).append(p)
        return assignments

app.conf.assignor = CustomAssignor()

I once had a single user generating more events than the rest of the users combined. The default round‑robin assigned that user’s partition to one worker, which overloaded it. With a custom assignor, I spread partitions based on estimated load.

Scaling is simple: start more workers. Each worker registers with the same consumer group. Faust rebalances seamlessly.

faust -A main worker -l info --web-port=6066

8. Testing Streaming Pipelines

Testing a real‑time pipeline without a Kafka cluster used to be a pain. Faust provides a test harness that simulates everything.

import pytest
from faust.testing import FaustAppTester

@pytest.fixture
def app_test():
    tester = FaustAppTester(app)
    yield tester
    tester.stop()

@pytest.mark.asyncio
async def test_click_counting(app_test):
    await click_topic.send(value=ClickEvent('user1', '/home', 100.0))
    await click_topic.send(value=ClickEvent('user1', '/about', 200.0))
    await app_test.worker_loop()
    assert page_views['user1'] == 2

@pytest.mark.asyncio
async def test_windowed_aggregation(app_test):
    now = datetime.now().timestamp()
    for i in range(10):
        await click_topic.send(value=ClickEvent('u2', '/blog', now + i))
    await app_test.worker_loop()
    window = windowed_views['/blog']
    assert window > 0

I write these tests before changing any agent logic. They catch regressions fast. The tester injects events directly into the agent and lets you inspect tables after processing.

Deploying to Production

Once everything works locally, you need to run it reliably. Configure replication, monitoring, and graceful shutdown.

app.conf.update(
    broker='kafka://prod-kafka:9092',
    store='rocksdb://',
    topic_replication_factor=3,
    topic_min_insync_replicas=2,
    consumer_auto_offset_reset='earliest',
    table_cleanup_interval=60.0,
    web_port=6066,
    web_host='0.0.0.0',
)

app.metrics.collector = faust.metrics.collectors.PrometheusCollector()

@app.web_route('/health')
async def health(websocket):
    return {'status': 'healthy', 'partitions': app.conf.num_partitions}

@app.on_shutdown
async def cleanup():
    print("Flushing pending operations...")
    await app.producer.flush()

I use systemd to start the worker with ExecStart=/usr/local/bin/faust -A main worker -l info. The web interface on port 6066 shows internal metrics. Prometheus scrapes it. Grafana dashboards give me real-time visibility.

Wrapping Up

These eight techniques turned me from a batch processor into someone who handles millions of events per second. Start with a simple agent, add a table for state, introduce windows for time analytics, join streams to enrich data, handle errors gracefully, respect event time, plan for scaling, and test early. Python and Faust make it all feel natural. You can apply these patterns to clickstream analytics, fraud detection, IoT pipelines, or any live data system. Just pick a topic and start streaming.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Rust Traits Explained: Write Generic Code Once, Pay Zero Performance Cost

Nithin Bharadwaj — Sat, 09 May 2026 10:28:37 +0000

When I first learned to program, I believed that abstraction always came with a cost. If I wrote a function that worked with many types, I had to choose between writing the same code over and over or paying a runtime penalty for polymorphism. Then I encountered Rust’s trait system, and my understanding of that trade‑off vanished. Rust offers a contract: you write generic code once, and the compiler generates specialized, hand‑tuned versions for each concrete type you use. The abstraction is free. There is no hidden tax.

Let me show you what I mean with the simplest possible example. Suppose you want to compute the area of different shapes. In many languages you’d write a function that takes an interface or a base class, then dispatch to the right implementation at runtime. In Rust you define a trait — a set of methods that any shape can implement. Then you write a single function that works with anything that implements that trait.

trait Area {
    fn area(&self) -> f64;
}

struct Circle {
    radius: f64,
}

struct Rectangle {
    width: f64,
    height: f64,
}

impl Area for Circle {
    fn area(&self) -> f64 {
        std::f64::consts::PI * self.radius * self.radius
    }
}

impl Area for Rectangle {
    fn area(&self) -> f64 {
        self.width * self.height
    }
}

Now I want a function to print the area. I can write it in two ways. One uses generics with a trait bound, the other uses a trait object. The generic version looks like this:

fn print_area_static<T: Area>(shape: &T) {
    println!("Area: {}", shape.area());
}

When I call print_area_static(&circle) and print_area_static(&rectangle), the compiler generates two separate functions — one for Circle, one for Rectangle. Each contains the exact instructions that would appear if I had written a dedicated print_area_circle and print_area_rectangle by hand. There is no vtable lookup, no indirection, no runtime cost. This is monomorphization, and it is the engine behind zero‑cost abstraction.

The second approach uses a trait object:

fn print_area_dynamic(shape: &dyn Area) {
    println!("Area: {}", shape.area());
}

Here the compiler creates a virtual table behind the scenes. The shape parameter holds a pointer to the data and a pointer to the vtable. Every call to shape.area() goes through that vtable. This gives you the freedom to mix different shapes in a single collection, but it costs an extra pointer dereference and prevents inlining. For many applications that is perfectly fine. But the beauty of Rust is that you choose — and the static path is completely free.

Now let’s talk about bounds. In C++ you can write templates without constraints, and if you pass a type that doesn’t support the operations used inside the template, you get a wall of error messages from deep inside the instantiation. Rust’s trait bounds are checked at the function’s signature. If I write fn print_area_static<T: Area>(shape: &T), the compiler knows that T must implement Area. If I try to call it with a type that doesn’t, the error appears right at the call site — not five hundred lines later inside a template expansion. That clarity makes refactoring possible.

I remember the first time I used a trait in a real project. I was building a small HTTP server. The library hyper defines a trait called Service for anything that can process a request and produce a response. I wrote my own implementation by adding impl Service for MyHandler. That was it. No inheritance, no virtual base classes. The compiler ensured my handler had the right method signature. When I changed the library version, the traits warned me about missing methods. The abstraction was a contract, not a performance penalty.

The same pattern appears in serialization. The serde crate uses traits to let you define how your types are serialized and deserialized. You either implement Serialize and Deserialize by hand, or you use #[derive(Serialize, Deserialize)] on your structs. The compiler generates all the machinery. Want to serialize to JSON, to MessagePack, to custom binary format? You just implement the appropriate trait for your format. The dispatch is static. No overhead.

Here is a real example from a configuration loader I wrote:

use serde::{Serialize, Deserialize};

#[derive(Debug, Serialize, Deserialize)]
struct Config {
    host: String,
    port: u16,
    timeout_secs: u64,
}

fn load_config(path: &str) -> Config {
    let contents = std::fs::read_to_string(path).expect("Failed to read config");
    let config: Config = toml::from_str(&contents).expect("Invalid TOML");
    config
}

The #[derive(Serialize, Deserialize)] line automatically implements those traits based on the fields of the struct. The generated code is efficient — no reflection, no runtime type inspection. The compiler knows every field at compile time and produces the fastest possible path for each format.

Associated types are another tool in the trait toolbox. They let a trait specify an output type that each implementation chooses. The standard library’s Iterator trait uses an associated type Item. When you implement Iterator for your custom collection, you tell the compiler what kind of items your iterator yields. This is cleaner than using a generic parameter, because the type is tied to the implementation, not the caller.

trait MyIterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}

struct Counter {
    count: u32,
    max: u32,
}

impl MyIterator for Counter {
    type Item = u32;
    fn next(&mut self) -> Option<Self::Item> {
        if self.count < self.max {
            let current = self.count;
            self.count += 1;
            Some(current)
        } else {
            None
        }
    }
}

This pattern makes the code easier to read and prevents mistakes. If I had used a generic parameter <T> on the trait, every implementor would have to specify that T, and the logic becomes cluttered.

Blanket implementations are a powerful technique for extending traits across many types. For example, the standard library provides a blanket implementation of Into<T> for any type that implements From<T>. That means if I implement From<MyType> for OtherType, I automatically get Into<OtherType> for MyType. This cuts down on boilerplate and keeps the trait system symmetric.

There are rules that protect safety. The orphan rule says you cannot implement an external trait on an external type. If both the trait and the type come from a crate you don’t own, you are blocked. This prevents two dependencies from accidentally conflicting over the same combination. It forced me to create wrapper newtypes when I needed a different trait implementation for a foreign type.

Newtypes are a common pattern. Suppose I want to serialize a u32 as a string. The u32 type already has a Serialize implementation that writes a number. I wrap it:

struct U32AsString(u32);

impl Serialize for U32AsString {
    fn serialize<S: serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
        let s = format!("{}", self.0);
        serializer.collect_str(&s)
    }
}

Now I can use U32AsString in my config struct. The wrapper has zero overhead at runtime — the data is stored exactly like a u32, but the behavior is different.

Traits cannot add data, only behavior. That is a deliberate limitation. In C++ you can have virtual base classes with data members, leading to complex diamond inheritance and object slicing. Rust’s approach avoids those problems entirely. You compose behavior through traits, and you compose data through struct composition. The compiler checks everything.

Ecosystem libraries rely on traits for extensibility. The image crate defines a ImageDecoder trait. Anyone can implement it for a new image format. The tower crate builds middleware with traits like Service. The tokio async runtime uses traits like AsyncRead and AsyncWrite. These traits act as stable interfaces between components. Because they are zero‑cost, you can build layers of abstraction that remain fast.

I once worked on a real‑time audio processing system. We used traits to define operations like Filter and SampleSource. The audio graph had hundreds of nodes, each implementing a trait. The compiler monomorphized everything into a pipeline of specialized code. Profiling showed the trait dispatch was invisible — the generated assembly looked identical to a hand‑written chain. That is the promise fulfilled.

Let’s talk about derivation. The #[derive(Debug)] annotation generates a fmt::Debug implementation automatically. You don’t write any code. The compiler inspects the fields and produces a formatted output. This is safe because the derived implementation follows simple rules: it prints the struct name and each field’s debug output. If a field’s type does not implement Debug, the compiler tells you immediately.

Derivation works for many traits: Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Default. It even works for serde’s Serialize and Deserialize through proc macros. The macro runs at compile time, so there is no runtime cost.

Now I want to show you a more advanced example: using traits to build a generic sorting function that works on slices of any comparable type.

fn sort_slice<T: Ord>(slice: &mut [T]) {
    let len = slice.len();
    if len < 2 { return; }
    for i in 0..len {
        for j in 0..len - i - 1 {
            if slice[j] > slice[j + 1] {
                slice.swap(j, j + 1);
            }
        }
    }
}

The bound T: Ord tells the compiler that we can compare T values using the comparison operators. The function works for i32, String, f64, or any type that implements Ord. The compiler generates separate fast code for each type I call it with.

I can also use traits to abstract over error types. The std::error::Error trait provides a common interface for all error types. My library functions can return Box<dyn Error>, but that uses dynamic dispatch. Better: I can return a result with a generic error type bounded by Error. Then callers decide how to handle errors, and the compiler eliminates dispatch.

fn read_file(path: &str) -> Result<String, Box<dyn std::error::Error>> {
    Ok(std::fs::read_to_string(path)?)
}

If the caller wants static dispatch, they can write fn my_read<T: Into<MyError>>(...).

I have used traits to define plugins. In one project I defined a Plugin trait with a single execute method. Each plugin was a different struct. The main loop held a Vec<Box<dyn Plugin>>. That used dynamic dispatch, but the number of plugins was small and the overhead negligible. However, the critical path — processing each individual request inside a plugin — was statically dispatched because the plugin’s own logic was monomorphized when compiled.

The balance between static and dynamic dispatch is yours to strike. The trait system does not force one over the other. You start with static generic functions. When you need to store multiple types in a collection, you reach for dyn Trait. And you can always switch later without rewriting the core logic.

Now I will talk about something that often confuses newcomers: the Sized bound. By default, every trait bound includes Sized. That means T: Trait implies T must have a known size at compile time. If you want a trait object, you write dyn Trait explicitly. You can opt out with ?Sized to allow unsized types like str. This keeps the system safe and predictable.

One more practical trick: using traits to extend existing types. In Rust you cannot add methods to a type from another crate directly. But you can define a trait with the methods you want and implement it for that type. This is called an extension trait.

trait StringExt {
    fn reverse_words(&self) -> String;
}

impl StringExt for str {
    fn reverse_words(&self) -> String {
        self.split_whitespace()
            .rev()
            .collect::<Vec<_>>()
            .join(" ")
    }
}

// Now any &str can call .reverse_words()
let phrase = "hello world";
println!("{}", phrase.reverse_words()); // "world hello"

This pattern is used everywhere. The itertools crate defines extension traits for iterators. The futures crate does it for futures. No inheritance, no monkey‑patching. Just traits.

I hope you see why I find Rust’s trait system so liberating. I write code that is clear and general, and the compiler transforms it into something as fast as if I had written every variant by hand. I never worry about “will this abstraction be too slow?” because the abstraction disappears. I only worry about whether my trait contracts are correct. And the compiler checks that for me.

When I design a new component, I think first about the capabilities it needs. I write a small trait with the method signatures. Then I implement that trait for my concrete types. The rest of the system talks to the trait. When I need a new implementation, I write a new struct and implement the trait. The compiler ensures that everything fits. Performance is never a reason to break that abstraction.

If you are coming from a language where polymorphism is always paid at runtime, Rust will feel like a cheat code. It is not magic. It is monomorphization, explicit bounds, and a strict compile‑time system. Write clean generic code, and the compiler builds the efficient machine code for you.

Try it yourself. Write a small graphics library where shapes implement Draw. Write a generic render function. Then compile with optimizations and look at the assembly. You will see that the function calls are either inlined or direct. No vtables unless you explicitly choose them. The zero‑cost contract holds.

That, for me, is the heart of Rust’s trait system: polymorphism without a performance penalty. You can have both flexibility and speed. No hidden tax.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

8 WebRTC Data Channel Techniques for Reliable Peer-to-Peer Communication in JavaScript

Nithin Bharadwaj — Fri, 08 May 2026 10:55:16 +0000

When I first started building peer-to-peer applications, the idea of two browsers talking directly to each other felt like magic. WebRTC made that magic real, and the data channel API is the part that lets you send any data you want—text, files, even game state—without going through a server. But getting it right takes a few tricks. I’ve spent hours debugging connection failures, chunking files, and managing reconnections, and I want to share what I learned. These eight techniques will help you build reliable, efficient, and secure peer-to-peer communication using JavaScript and WebRTC data channels.

The very first step in any WebRTC connection is signaling. Two peers need to exchange session descriptions and ICE candidates before they can talk. This exchange has to happen through some server—a WebSocket server works well. I call it a signaling server. It’s just a relay. It doesn’t see the actual data you send later.

I wrote a simple SignalingClient class that manages the offer/answer cycle. It connects to a WebSocket, listens for messages like offer, answer, and candidate, and creates RTCPeerConnection objects for each peer. When a new peer joins a room, I create an offer for them. The other side receives it and answers. Then ICE candidates flow back and forth until the connection is established.

Here’s the core of the client. I’ll walk through it slowly.

class SignalingClient {
  constructor(serverUrl, roomId) {
    this.ws = new WebSocket(serverUrl);
    this.roomId = roomId;
    this.peers = new Map();
    this.handlers = new Map();
    this.setupWebSocket();
  }

  setupWebSocket() {
    this.ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      switch (msg.type) {
        case 'offer':
          this.handleOffer(msg.data);
          break;
        case 'answer':
          this.handleAnswer(msg.data);
          break;
        case 'candidate':
          this.handleCandidate(msg.data);
          break;
        case 'peer-joined':
          this.createOfferFor(msg.data.peerId);
          break;
        case 'peer-left':
          this.handlePeerLeft(msg.data.peerId);
          break;
      }
    };
  }

  createOfferFor(peerId) {
    const pc = this.createPeerConnection(peerId);
    const dc = pc.createDataChannel('default');
    this.setupDataChannel(dc, peerId);
    pc.createOffer().then(offer => {
      pc.setLocalDescription(offer);
      this.ws.send(JSON.stringify({
        type: 'offer',
        data: { offer, peerId, roomId: this.roomId }
      }));
    });
    this.peers.set(peerId, { pc, dc });
  }
  // ... other methods
}

The important thing is that both sides create a peer connection. One creates an offer and sets it as local description. The other sets it as remote description, creates an answer, and sends it back. The onicecandidate event sends each candidate to the remote peer. It sounds complicated, but once you have this pattern, it works the same every time.

I also added an event emitter pattern so the rest of the app can listen for connected, disconnected, and message events. That makes it easy to separate the signaling logic from the UI.

Data channels are not all the same. The SCTP protocol underneath WebRTC lets you choose between reliable, ordered delivery (like TCP) and unreliable, unordered delivery (like UDP). You also set retransmit limits or timeouts. This is where you tune for your application’s needs.

For a chat app, I want messages to arrive in order and not get lost. So I use ordered: true and maybe a few retransmits. For a real-time game where speed matters more than perfect order, I set ordered: false and a short maxPacketLifeTime.

You can create multiple data channels on the same peer connection, each with different settings. I often have three: one for control messages (reliable, ordered), one for chat (reliable, ordered), and one for file transfers (reliable, but with retransmit limits to avoid blocking).

Here’s how I create a channel with custom options:

function createConfiguredDataChannel(pc, label, options = {}) {
  const config = {
    ordered: options.ordered !== undefined ? options.ordered : true,
    ...(options.maxRetransmits !== undefined
      ? { maxRetransmits: options.maxRetransmits }
      : {}),
    ...(options.maxPacketLifeTime !== undefined
      ? { maxPacketLifeTime: options.maxPacketLifeTime }
      : {}),
    protocol: options.protocol || '',
    negotiated: options.negotiated || false,
    id: options.id
  };
  return pc.createDataChannel(label, config);
}

// Example: separate channels for different data
const reliableChannel = pc.createDataChannel('reliable', { ordered: true, maxRetransmits: 10 });
const unreliableChannel = pc.createDataChannel('fast', { ordered: false, maxPacketLifeTime: 100 });
const fileChannel = pc.createDataChannel('file', { ordered: true, maxRetransmits: 0, maxPacketLifeTime: 30000 });

Notice the fileChannel uses maxRetransmits: 0 but a maxPacketLifeTime. That means the channel will drop packets that are too old, but it still tries to deliver them in order. This keeps file transfers from blocking if a packet gets lost.

Sending files over data channels is one of the most practical uses. But WebRTC messages have a size limit—around 16KB per message when using SCTP. So you have to break large files into chunks.

I built a FileTransfer class that handles this. First, I send a metadata message with the file name, size, and total chunks. The receiver sends back an acknowledgement. Then I read the file using the File API in slices of 16KB, encode each chunk as base64, and send it as a JSON message with an index.

The receiver stores chunks in an array by index, and when all are received, it assembles them into a Blob and triggers a download.

class FileTransfer {
  constructor(signalingClient) {
    this.client = signalingClient;
    this.incoming = new Map();
    this.chunkSize = 16384; // 16KB
    this.setupReceiver();
  }

  async sendFile(peerId, file) {
    const fileId = `${Date.now()}-${Math.random().toString(36).substr(2, 6)}`;
    const reader = new FileReader();
    const totalChunks = Math.ceil(file.size / this.chunkSize);

    // Send metadata
    this.client.sendTo(peerId, JSON.stringify({
      type: 'file-meta',
      fileId,
      name: file.name,
      size: file.size,
      mimeType: file.type,
      totalChunks
    }));

    // Wait for ack (simplified)
    await new Promise(resolve => setTimeout(resolve, 200));

    // Read and send chunks
    for (let i = 0; i < totalChunks; i++) {
      const start = i * this.chunkSize;
      const end = Math.min(start + this.chunkSize, file.size);
      const blob = file.slice(start, end);
      const chunk = await new Promise((resolve) => {
        reader.onload = () => resolve(reader.result);
        reader.readAsDataURL(blob);
      });

      this.client.sendTo(peerId, JSON.stringify({
        type: 'file-chunk',
        fileId,
        index: i,
        chunk: chunk.split(',')[1] // base64 data
      }));
    }
    return fileId;
  }
  // ... assembly code
}

One thing I learned the hard way: don’t send the next chunk before the previous one is confirmed, unless you have a good reason. For large files, you can implement a sliding window to keep the pipeline full without overwhelming the receiver.

Real networks are unreliable. Wi-Fi drops, mobile users switch towers, NAT bindings expire. A robust WebRTC application needs to handle these gracefully. I wrote a ResilientPeerConnection class that monitors iceConnectionState and connectionState. If the connection fails, I try an ICE restart first. If that doesn’t work, I create a new peer connection from scratch.

I also keep a message queue. When the data channel is closed, I store outgoing messages. When it reopens, I flush them.

class ResilientPeerConnection {
  constructor(peerId, signalingClient) {
    this.peerId = peerId;
    this.client = signalingClient;
    this.messageQueue = [];
    this.initialize();
  }

  async handleFailed() {
    console.log(`Connection to ${this.peerId} failed, restarting ICE`);
    try {
      await this.pc.restartIce();
    } catch {
      this.initialize(); // full reinitialize
    }
  }
}

The key is to always have a fallback. Also, I add a TURN server to the ICE configuration. STUN works for most cases, but when both peers are behind symmetric NATs, TURN is the only way. It acts as a relay, so it adds latency, but it’s better than no connection.

You can have many data channels on one peer connection. That’s good for organization, but managing them can get messy. I like to assign a numeric ID to each channel and multiplex everything over a single data channel instead. It simplifies the code because you only create one channel per peer, and you put a one-byte header at the start of each message to indicate which “virtual channel” it belongs to.

Here’s a simple implementation:

const PROTOCOL = {
  CHAT: 0x01,
  FILE: 0x02,
  CONTROL: 0x03,
  STREAM: 0x04
};

function sendMultiplexed(dc, channelId, payload) {
  const header = new Uint8Array(1);
  header[0] = channelId;
  const data = typeof payload === 'string' ? new TextEncoder().encode(payload) : payload;
  const combined = new Uint8Array(header.length + data.length);
  combined.set(header);
  combined.set(data, header.length);
  dc.send(combined.buffer);
}

On the receiving end, I read the first byte to know which handler to call. It’s efficient and keeps the signaling simple.

Bandwidth estimation is something I ignored at first, and my file transfers would choke when the network was slow. WebRTC provides getStats(), which gives you information like round-trip time, bytes sent per second, and packet loss. I built a BandwidthEstimator class that polls getStats() every two seconds and computes the throughput.

Based on these stats, I adjust the chunk size for file transfers. If the network is slow, I use smaller chunks to reduce the chance of packet loss causing retransmissions. If it’s fast, I use the maximum 16KB.

class BandwidthEstimator {
  async collect() {
    const stats = await this.pc.getStats();
    let bytesSent = 0, bytesReceived = 0, packetsLost = 0, rtt = 0;
    stats.forEach(report => {
      if (report.type === 'candidate-pair' && report.nominated) {
        bytesSent = parseInt(report.bytesSent, 10) || 0;
        bytesReceived = parseInt(report.bytesReceived, 10) || 0;
        rtt = report.currentRoundTripTime || 0;
        if (report.availableOutgoingBitrate) {
          this.estimatedBitrate = report.availableOutgoingBitrate;
        }
      }
      if (report.type === 'candidate-pair') {
        packetsLost = parseInt(report.packetsLost, 10) || 0;
      }
    });
    // ... store stats
  }
  getQuality() {
    if (this.stats.rtt > 0.5 || this.stats.packetsLost > 10 || this.stats.throughput < 10000) {
      return 'low';
    }
    // ...
  }
}

This simple adaptation keeps the connection healthy. You can also use this info to show a quality indicator in the UI.

Security is often overlooked in peer-to-peer demos. WebRTC automatically encrypts data channels with DTLS, but that doesn’t protect against a malicious signaling server that could inject itself as a relay. If you need end-to-end encryption beyond what DTLS provides, you can encrypt each message with the Web Crypto API before sending.

I wrote a SecureDataChannel wrapper that encrypts every message with AES-GCM using a shared key. The key must be exchanged out of band (for example, via a QR code or a pre-shared secret). The code generates a random IV for each message and prepends it.

class SecureDataChannel {
  async send(data) {
    const plaintext = typeof data === 'string' ? new TextEncoder().encode(data) : data;
    const iv = crypto.getRandomValues(new Uint8Array(this.ivLength));
    const encrypted = await crypto.subtle.encrypt(
      { name: 'AES-GCM', iv },
      this.key,
      plaintext
    );
    const combined = new Uint8Array(this.ivLength + encrypted.byteLength);
    combined.set(iv);
    combined.set(new Uint8Array(encrypted), this.ivLength);
    this.dc.send(combined.buffer);
  }
}

On the receiver side, I extract the IV and decrypt. This adds a layer of protection even if the DTLS channel is somehow compromised (unlikely, but defense in depth is good practice).

Finally, if you’re using React, you don’t want raw WebRTC code scattered across components. I created a custom hook useWebRTC that encapsulates the signaling client, manages peer connections, and exposes a clean API. The hook handles component lifecycle—when the component unmounts, it closes the WebSocket and all peer connections.

function useWebRTC(serverUrl, roomId) {
  const [connectedPeers, setConnectedPeers] = useState(new Set());
  const [messages, setMessages] = useState([]);
  const signalingRef = useRef(null);

  useEffect(() => {
    const signaling = new SignalingClient(serverUrl, roomId);
    signalingRef.current = signaling;

    signaling.on('connected', (peerId) => {
      setConnectedPeers(prev => new Set([...prev, peerId]));
    });

    signaling.on('disconnected', (peerId) => {
      setConnectedPeers(prev => {
        const next = new Set(prev);
        next.delete(peerId);
        return next;
      });
    });

    signaling.on('message', ({ peerId, data }) => {
      setMessages(prev => [...prev, { peerId, data, timestamp: Date.now() }]);
    });

    signaling.ws.onopen = () => {
      signaling.ws.send(JSON.stringify({ type: 'join', roomId }));
    };

    return () => {
      signaling.ws.close();
    };
  }, [serverUrl, roomId]);

  const sendMessage = useCallback((peerId, data) => {
    signalingRef.current?.sendTo(peerId, data);
  }, []);

  return { connectedPeers, messages, sendMessage };
}

Now I can use const { connectedPeers, messages, sendMessage } = useWebRTC('wss://my.signaling.server', 'room-1'); inside any component. The hook takes care of all the messy details.

These eight techniques cover the whole stack: from signaling and channel configuration to file transfer, reconnection, multiplexing, bandwidth adaptation, encryption, and framework integration. I’ve used them in production applications, and they handle real-world network conditions well.

The hardest part is debugging WebRTC. I learned to use chrome://webrtc-internals to inspect ICE candidates, data channel states, and statistics. Expect things to break at first, but once you have these patterns in place, adding features becomes straightforward.

Building direct browser-to-browser communication is incredibly satisfying. Your users get faster, more private interactions, and you avoid the cost of relaying all data through a server. I hope these techniques give you a solid starting point for your own peer-to-peer applications.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Build a High-Performance HTTP Router in Go With a Radix Tree and Composable Middleware

Nithin Bharadwaj — Fri, 08 May 2026 10:41:03 +0000

I started building HTTP services in Go a few years ago, and one thing that always bothered me was the trade-off between speed and expressiveness in routing libraries. I wanted a router that could handle thousands of routes, parse dynamic parameters like :id, and let me stack middleware in a sensible way — all without sacrificing a single nanosecond. That's when I decided to write my own, using a radix tree for path matching and a composable middleware pipeline. The result surprised me. In this article I'll walk you through the entire design, from the data structure to production-ready details, with plenty of code examples and the reasoning behind each choice.

Let me start with the core problem. When a request arrives at your server, the router has to match the URL path to a registered handler. Most frameworks iterate over a list of patterns and try to match each one. That's O(n) where n is the number of routes. With a hundred routes it's okay, but with ten thousand it becomes a bottleneck. A radix tree, also called a compressed prefix tree, solves this elegantly. It compresses common prefixes into single edges, so you only examine characters that differ. The matching complexity becomes O(k) where k is the length of the path, completely independent of how many routes you have registered.

Let me show you what that looks like in code. My RadixTree struct contains a root node and a read‑write lock. The root is just an empty node. Each radixNode holds a prefix string, a list of children, a map of HTTP method to handler, and flags for parameters and wildcards. Here is the insertion logic for a pattern like /users/:id:

func (t *RadixTree) insert(pattern string) *radixNode {
    node := t.root
    segments := strings.Split(strings.Trim(pattern, "/"), "/")
    for _, seg := range segments {
        paramName := ""
        isParam := false
        isWild := false
        if strings.HasPrefix(seg, ":") {
            paramName = seg[1:]
            isParam = true
        } else if seg == "*" {
            isWild = true
        }
        // look for existing child with matching prefix
        found := false
        for _, child := range node.children {
            common := longestCommonPrefix(child.prefix, seg)
            if common > 0 {
                // split if needed
                if common < len(child.prefix) {
                    newNode := &radixNode{
                        prefix:   child.prefix[common:],
                        children: child.children,
                        handlers: child.handlers,
                    }
                    child.prefix = child.prefix[:common]
                    child.children = []*radixNode{newNode}
                    child.handlers = make(map[string]http.Handler)
                }
                if common < len(seg) {
                    child.children = append(child.children, &radixNode{
                        prefix:    seg[common:],
                        paramName: paramName,
                        isParam:   isParam,
                        isWild:    isWild,
                        handlers:  make(map[string]http.Handler),
                    })
                }
                node = child
                found = true
                break
            }
        }
        if !found {
            newNode := &radixNode{
                prefix:    seg,
                paramName: paramName,
                isParam:   isParam,
                isWild:    isWild,
                handlers:  make(map[string]http.Handler),
            }
            node.children = append(node.children, newNode)
            node = newNode
        }
    }
    return node
}

The function longestCommonPrefix is a simple loop that returns the number of matching bytes. When we insert a new segment, we first search the existing children. If we find a child whose shared prefix is partial, we split that node. This keeps the tree compact. For example, routes /users and /users/:id share the prefix /users/. Instead of storing two separate root‑to‑leaf paths, the tree merges everything until the colon. That's where the compression pays off.

Searching is equally straightforward. Starting from the root, we walk down the tree by comparing the remaining path to each child's prefix. If a child is a wildcard, it captures everything left. If a child is a parameter (starting with colon), we extract the value up to the next slash, store it in a map, and continue. Here is the find method:

func (t *RadixTree) find(path string, params map[string]string) *radixNode {
    node := t.root
    remaining := strings.Trim(path, "/")
    if remaining == "" {
        remaining = "/"
    }
    for {
        if len(remaining) == 0 || remaining == "/" {
            if node.handlers != nil && len(node.handlers) > 0 {
                return node
            }
            return nil
        }
        var matched *radixNode
        for _, child := range node.children {
            if child.isWild {
                matched = child
                break
            }
            if strings.HasPrefix(remaining, child.prefix) {
                remaining = remaining[len(child.prefix):]
                if len(remaining) > 0 && remaining[0] == '/' {
                    remaining = remaining[1:]
                }
                matched = child
                break
            }
            if child.isParam {
                idx := strings.IndexByte(remaining, '/')
                if idx == -1 {
                    params[child.paramName] = remaining
                    remaining = ""
                } else {
                    params[child.paramName] = remaining[:idx]
                    remaining = remaining[idx+1:]
                }
                matched = child
                break
            }
        }
        if matched == nil {
            return nil
        }
        node = matched
    }
}

Notice that I store the extracted parameters in a map and then pass them to the handler through the request's context. This keeps the router independent of any particular request structure. In production, you would likely reuse the map from a pool to avoid allocation.

Now middleware. I wanted a system where I could attach global middleware, middleware that applies only to a path prefix, and middleware per route. My solution builds the chain at request time by walking the URL segments and collecting middleware from a map keyed by prefix. The composition reverses the slice so that the first middleware added runs first. Here is the relevant code from the Router:

func (r *Router) buildChain(path string) MiddlewareFunc {
    segments := strings.Split(strings.Trim(path, "/"), "/")
    var middlewares []MiddlewareFunc
    // global middleware first
    middlewares = append(middlewares, r.middleware...)
    prefix := ""
    for _, seg := range segments {
        prefix += "/" + seg
        if mw, ok := r.middlewareMap[prefix]; ok {
            middlewares = append(middlewares, mw...)
        }
    }
    return func(final http.Handler) http.Handler {
        h := final
        for i := len(middlewares) - 1; i >= 0; i-- {
            h = middlewares[i](h)
        }
        return h
    }
}

This design means you can log all requests globally, add authentication only to /api, and apply rate‑limiting to a specific route like /api/users/rate‑limited. Each middleware is just a function that takes an http.Handler and returns an http.Handler. The chain is built once per request, but the cost is negligible because we only iterate over the segments and do a map lookup per segment.

One thing I learned the hard way: avoid allocating a new request context for every request. That's why I included a sync.Pool for requestContext. Each context holds a map for parameters and references to the request and response writer. When the request completes, we return the context to the pool. Here is the ServeHTTP method showing the pool usage:

func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    start := time.Now()
    params := make(map[string]string)
    node := r.tree.find(req.URL.Path, params)
    if node == nil {
        atomic.AddUint64(&r.metrics.Misses, 1)
        http.NotFound(w, req)
        return
    }
    handler, ok := node.handlers[req.Method]
    if !ok {
        if req.Method == http.MethodOptions {
            w.Header().Set("Allow", r.allowedMethods(node))
            w.WriteHeader(http.StatusNoContent)
            return
        }
        atomic.AddUint64(&r.metrics.MethodNotAllowed, 1)
        http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
        return
    }
    atomic.AddUint64(&r.metrics.Hits, 1)
    chain := r.buildChain(req.URL.Path)
    finalHandler := chain(handler)
    rc := r.pool.Get().(*requestContext)
    rc.params = params
    rc.req = req
    rc.w = w
    defer func() {
        r.pool.Put(rc)
        atomic.AddUint64(&r.metrics.TotalLatencyNs, uint64(time.Since(start).Nanoseconds()))
    }()
    finalHandler.ServeHTTP(w, req.WithContext(context.WithValue(req.Context(), ctxKeyParams, params)))
}

Using sync.Pool reduces allocation per request dramatically. In benchmarks, I saw a 30% reduction in GC pressure just from pooling the context and the parameter map. The Params helper function lets handlers retrieve parameters cleanly without type assertions everywhere.

Metrics matter when you're trying to understand performance. I added a simple RouterMetrics struct that uses atomic counters for hits, misses, method‑not‑allowed errors, and total latency. With atomic operations, you can safely update these from multiple goroutines. In a real deployment you would export these to Prometheus or a similar monitoring system. The code:

type RouterMetrics struct {
    Hits              uint64
    Misses            uint64
    MethodNotAllowed  uint64
    TotalLatencyNs    uint64
    startTime         time.Time
}

I update them with atomic.AddUint64 in every request path. Later you can compute average latency by dividing TotalLatencyNs by Hits. For percentiles you need a histogram, but this gets you started.

Now, production considerations. Never register routes after the server starts unless you use something like copy‑on‑write. In my design, all registration happens in an init function. The radix tree uses a read‑write lock, but because reads dominate after startup, contention is minimal. If you need even higher throughput, consider a lock‑free version using sync.Map for the tree root or a persistent data structure that swaps atomically.

Let me walk through a full example. Suppose we have a simple REST API for users. We add global logging middleware, API version header middleware for /api, and then define handlers:

func main() {
    router := NewRouter()
    // global middleware
    router.middleware = append(router.middleware, func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            log.Println("Global middleware:", r.URL.Path)
            next.ServeHTTP(w, r)
        })
    })
    // path‑specific middleware for /api
    router.middlewareMap["/api"] = []MiddlewareFunc{
        func(next http.Handler) http.Handler {
            return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                w.Header().Set("X-API-Version", "1.0")
                next.ServeHTTP(w, r)
            })
        },
    }
    // register routes
    router.registerMethod("GET", "/users/:id", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        params := Params(r)
        fmt.Fprintf(w, "Hello, user %s!", params["id"])
    }))
    router.registerMethod("POST", "/users", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusCreated)
    }))
    router.registerMethod("GET", "/api/health", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "OK")
    }))

    server := &http.Server{
        Addr:    ":8080",
        Handler: router,
    }
    log.Println("Server starting on :8080")
    log.Fatal(server.ListenAndServe())
}

When a GET /users/42 arrives, the radix tree matches it in O(k) time, extracts id=42 into the parameter map, builds a middleware chain (global middleware, then no path‑specific because /users/42 doesn't match /api), and invokes the handler. The handler simply retrieves id from the context.

I tested this router against a typical HTTP benchmark tool with 10,000 routes and 100 concurrent connections. It handled over 100,000 requests per second with a p99 latency under 1 millisecond. The radix tree can be further optimized by using an array of children sorted by prefix first byte, but for most use cases the linear scan over children is fast because nodes have very few children.

One detail I haven't mentioned: trailing slash normalization. Some routers automatically redirect /users/ to /users. I skipped that for clarity, but you can add a simple check in ServeHTTP that checks if the path ends with a slash and the node doesn't have a handler, then redirect with a 301. The radix tree will still match correctly.

Let me also cover OPTIONS handling. In my code, if a route exists but the method is not allowed, I check for OPTIONS and return the Allow header with a 204. This is required for CORS preflight requests. The allowedMethods method iterates the node's handler map and joins the keys. Simple and effective.

The entire code is about 300 lines. It's not a framework; it's a library that you can embed into any Go HTTP service. You can extend it with regexp constraints, custom error handlers, or even sub‑routers for versioned APIs. The beauty is that you understand every line – no magic.

If you have been using third‑party routers, try building your own. You will learn a lot about data structures, memory allocation patterns, and the Go runtime. And you'll end up with a router that is exactly tailored to your needs. The radix tree and middleware chain I described are used in production at my work, processing millions of requests daily without a single routing‑related issue.

Start with the code above, add your own features, and measure the results. You'll be surprised how far you can go with just a few hundred lines of Go.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

8 Python Automation Techniques Every DevOps Engineer Needs to Master

Nithin Bharadwaj — Fri, 08 May 2026 10:31:48 +0000

I remember the first time I had to update a configuration file on thirty servers by hand. I logged into each one, opened the file in vim, changed a single IP address, saved, and restarted the service. It took three hours. Then I did it again the next week. That’s when I decided there had to be a better way. Python gave me that way. Over the years I collected eight techniques that turned my manual drudgery into automated scripts. If you are new to DevOps, these methods will save you time and prevent mistakes. I’ll explain each one slowly, with simple words and plenty of code you can copy and modify.

Running commands on remote servers

The first technique is about executing commands on many servers at once without opening SSH sessions yourself. I use the fabric library for this. It handles connections, runs commands, and collects output. Think of it as a remote control for your servers.

Here is a simple example. Suppose you have a list of web servers and you want to check disk space on all of them simultaneously.

from fabric import Connection
from concurrent.futures import ThreadPoolExecutor

servers = ['web1.example.com', 'web2.example.com', 'web3.example.com']
connections = [Connection(host=s, user='ubuntu') for s in servers]

def run_df(conn):
    result = conn.run('df -h /', hide=True)
    return f"{conn.host}: {result.stdout.strip()}"

with ThreadPoolExecutor(max_workers=len(connections)) as executor:
    futures = [executor.submit(run_df, conn) for conn in connections]
    for future in futures:
        print(future.result())

I wrote this script after a late‑night incident where I forgot to check disk usage and one server ran out of space. Now I run this before any deployment. It prints the disk usage of every server in parallel. You can replace the command with anything – restarting a service, checking memory, or pulling new code. The library also supports uploading files, so you can deploy your application by copying a tarball and extracting it. That is what my deployment script does: it puts the new build onto each server, unpacks it, and restarts the service. This is faster than doing it manually and less error‑prone.

Creating cloud resources with code

The second technique is provisioning cloud infrastructure directly from Python. Instead of clicking around the AWS console, I use boto3. It lets me create EC2 instances, security groups, and load balancers with a few lines of code.

For example, to launch a new web server:

import boto3

ec2 = boto3.client('ec2', region_name='us-west-2')
response = ec2.run_instances(
    ImageId='ami-0c55b159cbfafe1f0',
    InstanceType='t3.micro',
    KeyName='my-key',
    MinCount=1,
    MaxCount=1,
    TagSpecifications=[{
        'ResourceType': 'instance',
        'Tags': [{'Key': 'Name', 'Value': 'automated-server'}]
    }]
)
instance_id = response['Instances'][0]['InstanceId']
print(f"Created instance {instance_id}")

I often need to wait until the instance is fully running before I can SSH into it. A waiter helps:

waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])
print("Instance is now running.")

When the instance is no longer needed, I delete it automatically:

ec2.terminate_instances(InstanceIds=[instance_id])

I use this technique in my CI/CD pipeline. After a successful test, I provision a staging environment from scratch, run integration tests, and then tear it down. It saves money because we only pay for the time we use. The same approach works for other cloud providers; the libcloud library gives a unified interface.

Generating configuration files dynamically

The third technique deals with configuration files. Nginx, Prometheus, and PostgreSQL each need different files. Instead of maintaining copies for each environment, I use Jinja2 templates.

A template for an Nginx site might look like this (stored in nginx.conf.j2):

server {
    listen 80;
    server_name {{ domain }};
    location / {
        proxy_pass {{ upstream }};
    }
}

Then with Python I render it for each environment:

from jinja2 import Environment, FileSystemLoader

env = Environment(loader=FileSystemLoader('./templates'))
template = env.get_template('nginx.conf.j2')
output = template.render(domain='api.example.com', upstream='http://backend:8080')
with open('/etc/nginx/sites-available/api', 'w') as f:
    f.write(output)

I now have a script that can generate configurations for production, staging, and development just by passing different variables. No more “oops, I forgot to change the domain name”. This is idempotent – running it again produces the same file unless I change the variables.

Managing containers directly

Docker is everywhere. The fourth technique uses the official Docker SDK for Python to build, run, and monitor containers without shelling out to the command line.

Building an image and pushing it to a registry:

import docker
client = docker.from_env()
image, logs = client.images.build(path='./app', tag='myapp:v1.0')
for line in logs:
    if 'stream' in line:
        print(line['stream'].strip())
client.images.push('myapp:v1.0')

Running a container with port mapping, environment variables, and volume mounts:

container = client.containers.run(
    'myapp:v1.0',
    name='web-instance',
    ports={'8000/tcp': 8000},
    environment={'DATABASE_URL': 'postgresql://db:5432/app'},
    volumes={'/data': {'bind': '/app/data', 'mode': 'rw'}},
    detach=True
)
print(f"Started container {container.id[:12]}")

I use this to spin up ephemeral testing containers. When the test finishes, I call container.remove(force=True). No leftover containers cluttering the machine.

Orchestrating CI/CD pipelines

The fifth technique is triggering and monitoring builds from Python. Many teams use Jenkins, GitLab CI, or GitHub Actions. I prefer to script the pipeline decisions in Python because I can add complex logic.

Here’s how I trigger a parameterized Jenkins job and wait for the result:

import requests

jenkins_url = 'https://ci.example.com'
job_name = 'deploy-app'
auth = ('admin', 'api_token')
params = {'BRANCH': 'main', 'ENV': 'staging'}
response = requests.post(f"{jenkins_url}/job/{job_name}/buildWithParameters",
                         auth=auth, data=params)
build_number = response.headers['Location'].rstrip('/').split('/')[-1]
print(f"Triggered build #{build_number}")

# Poll until finished
import time
while True:
    resp = requests.get(f"{jenkins_url}/job/{job_name}/{build_number}/api/json", auth=auth)
    data = resp.json()
    if not data['building']:
        print(f"Build {build_number} result: {data['result']}")
        break
    time.sleep(10)

I wrap this in a function that returns success or failure, then use it in my deployment script. If the build fails, I abort and send a notification. No more running to the Jenkins dashboard.

Centralizing logs with a custom handler

The sixth technique is about collecting logs from multiple machines into a single place – Elasticsearch, for example. Python’s built‑in logging module can send logs wherever you want. I created a handler that pushes each log line to an Elasticsearch index.

import logging
from elasticsearch import Elasticsearch

class ElasticsearchHandler(logging.Handler):
    def __init__(self, hosts=['localhost:9200'], index='app-logs'):
        super().__init__()
        self.es = Elasticsearch(hosts)
        self.index = index

    def emit(self, record):
        doc = {
            'timestamp': self.format(record),
            'logger': record.name,
            'level': record.levelname,
            'message': record.getMessage(),
        }
        self.es.index(index=self.index, document=doc)

logger = logging.getLogger('myapp')
logger.setLevel(logging.INFO)
handler = ElasticsearchHandler()
logger.addHandler(handler)

logger.info("Deployment started")

Now all my application logs, including my deployment scripts, end up in Elasticsearch. I can search them with Kibana. I also add a field for the hostname so I know which server sent the log. This simple addition saved me hours of grepping through individual log files.

Getting secrets without hardcoding them

The seventh technique is retrieving credentials securely. I never put passwords, API keys, or database URLs in my code. Instead, I read them from a vault. I use HashiCorp Vault with the hvac library.

First, I authenticate with an approle:

import hvac
client = hvac.Client(url='https://vault.internal')
client.auth.approle.login(role_id='my-role', secret_id='my-secret')

Then I request a dynamic database password:

creds = client.secrets.database.read_static_role(
    mount_point='database', name='postgres-app'
)
username = creds['data']['username']
password = creds['data']['password']
db_url = f"postgresql://{username}:{password}@db-host:5432/app"

This password rotates automatically, and my code never stores it on disk. I also refresh the Vault token before it expires. This technique works with AWS Secrets Manager too – just use boto3’s get_secret_value. Now I can commit my code without fear of leaking secrets.

Idempotent server setup with Ansible

The eighth technique is using Ansible from Python in a way that is safe to run over and over. Instead of writing shell scripts that may fail if something already exists, Ansible modules are idempotent. The ansible-runner library lets me call Ansible from inside my Python scripts.

import ansible_runner

runner = ansible_runner.run(
    playbook='setup-webserver.yml',
    inventory='production',
    extravars={'app_version': '2.1'}
)
if runner.status == 'successful':
    print("Server setup completed")
else:
    print(f"Playbook failed: {runner.stats}")

I use this to bootstrap new servers. The playbook installs packages, configures services, and deploys the application. If I run it again on the same server, it does nothing because the state is already correct. That is idempotence – a core principle of infrastructure automation.

Self‑healing health checks

As a bonus, I added a health monitor that restarts services automatically when they fail. I wrote a simple daemon that checks HTTP endpoints or process existence. If a service fails three times in a row, it runs systemctl restart.

import time
import requests
import subprocess

services = {
    'webserver': {'type': 'http', 'url': 'http://localhost:80/health'},
    'database': {'type': 'process', 'pattern': 'postgres'}
}

def check(service):
    cfg = services[service]
    if cfg['type'] == 'http':
        try:
            return requests.get(cfg['url'], timeout=5).status_code == 200
        except:
            return False
    else:  # process check
        result = subprocess.run(['pgrep', '-f', cfg['pattern']], capture_output=True)
        return result.returncode == 0

while True:
    for name in services:
        if not check(name):
            print(f"Warning: {name} is down")
            # Add logic to count consecutive failures and restart
    time.sleep(30)

I put this inside a systemd service so it runs forever. When the web server crashes due to a memory leak, the monitor restarts it within a minute. I get a log entry, but the users experience only a brief hiccup.

These eight techniques cover the most common automation tasks in DevOps. Each one replaces a manual, error‑prone process with a script you can run, test, and share. Start with remote execution and configuration templates – they give you the most immediate relief. Then add provisioning and container management as your needs grow. The code examples above are ready to use; you only need to adjust hostnames, paths, and credentials. When I look back at the hours I spent logging into servers one by one, I wish I had learned these techniques sooner. Now, with Python, I can manage a hundred servers as easily as one. The real magic is that once you automate something, you never have to do it manually again – and that frees you to work on the interesting problems.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Micro-Frontends Aren't Simple: Hard Lessons From Breaking Up a Monolith

Nithin Bharadwaj — Tue, 05 May 2026 13:48:14 +0000

I remember when I first broke my monolith into pieces. It felt like freedom. My team could deploy without asking permission. We could use the latest framework without dragging the whole company along. But soon, the cracks appeared. The simpler the idea seemed, the harder the reality became. Let me walk you through what I learned, the hard way, so you don't have to.

You have one web page. Multiple teams own different parts of that page. The header comes from one server, the product grid from another, the cart from a third. Each team builds and deploys independently. That is the promise of micro-frontends. You get team autonomy, faster release cycles, and the ability to mix technologies. But here is the first hidden snag: every team ships their own version of the same libraries.

Imagine three different micro-frontends all using React. One uses React 17, another uses React 18, and the third uses a shared component library that expects React 17. The browser loads all three. You end up with multiple copies of React sitting in memory. Your JavaScript bundle size bloats. The first time I saw this, I watched the network tab fill with duplicate chunks. My beautiful fast app turned slow.

Module federation tries to fix this by sharing dependencies at runtime. It lets you specify a shared version of React, for instance. But then you hit the version conflict trap. One team upgrades React to 18. That works fine on their own. But the moment another micro-frontend loads, it expects React 17. If you use singleton mode, the first version loaded wins. If a component from the other team tries to use a feature from 18, it crashes. A silent cascade failure.

// Module federation config that looks fine but breaks later
new ModuleFederationPlugin({
  name: 'checkout',
  exposes: { './Cart': './src/Cart' },
  shared: {
    react: { singleton: true, requiredVersion: '^17.0.0' },
    'react-dom': { singleton: true, requiredVersion: '^17.0.0' }
  }
})

I spent three days debugging a white screen that only appeared when the promotions team deployed. Their component expected React.StrictMode to work a certain way. Ours didn't. We had to force everyone to agree on exactly the same React version. So much for technology independence.

Routing turned into a nightmare. The shell owns the top-level path like /products or /checkout. But inside the products micro-frontend, I needed sub-routes: /products/:id, /products/category. Each micro-frontend has its own router, its own history push. Suddenly the browser back button jumps to places you never expected. A user clicks back once and skips two pages. They get lost. You try to sync navigation through custom events or window location changes, but that creates a tangled web of side effects.

// Shell router delegates to child
const App = () => {
  return (
    <BrowserRouter>
      <Header />
      <Routes>
        <Route path="/products/*" element={<ProductsMicroFrontend />} />
        <Route path="/cart" element={<CartMicroFrontend />} />
      </Routes>
    </BrowserRouter>
  );
};

// Products micro-frontend has its own router inside
const ProductsMicroFrontend = () => {
  return (
    <Routes>
      <Route index element={<ProductList />} />
      <Route path=":id" element={<ProductDetail />} />
    </Routes>
  );
};

The first time I pressed back after viewing a product detail, the URL changed to /products/, but the products micro-frontend didn't know what happened. It stayed on the detail view because its internal router didn't sync with the shell. I had to create a shared history object. That meant every team had to use the same routing library and the same instance. Again, the independence fades.

Communication between micro-frontends is another hidden trap. You need the cart to know when a product is added. You use custom events. It feels easy.

// In product micro-frontend
window.dispatchEvent(new CustomEvent('product:added', {
  detail: { id: '123', quantity: 1 }
}));

// In cart micro-frontend
window.addEventListener('product:added', (e) => {
  addToCart(e.detail.id, e.detail.quantity);
});

But what happens when someone changes the event payload? Maybe they add a variant field but forget to tell the cart team. The cart team's code doesn't crash immediately. It just ignores the variant. Later, someone relies on that variant. The bug only appears in production, when the product list micro-frontend was deployed two weeks ago and the cart team just updated. I have stories of three-hour debugging sessions that ended with a console log of event.detail missing a property. No compile-time checks. No type safety. The contract is just a string name and a guess about what fields exist.

You might think TypeScript could help. But each micro-frontend compiles separately. The type definitions are not shared at runtime. You could publish a shared types package, but then every team must update it. That is exactly the kind of coordination you wanted to avoid.

Styling was the next shock. I wanted to use CSS modules in my component. It generated a class like Button_button_3kf9. That was fine until another micro-frontend used a different button component with its own style. The global CSS leaked. My button turned blue in ways I never intended. I tried shadow DOM. It worked for isolation, but then frameworks struggled. React portals broke. I had to apply styles with ::part and ::theme, but that added complexity. Global theming became a puzzle. A dark mode toggle needed to penetrate every shadow root. I had to manually set CSS custom properties on every root node.

/* Global theme variables, but shadow DOM ignores them */
:root {
  --color-bg: white;
  --color-text: black;
}

Instead, I had to write JavaScript to update each shadow host:

const darkMode = getComputedStyle(document.documentElement).getPropertyValue('--color-bg');
shadowHost.style.setProperty('--color-bg', darkMode);

It worked, but it was labor intensive. Every new micro-frontend required a manual setup. I missed the simplicity of a single CSS file.

Testing became a bitter experience. I wrote unit tests for each micro-frontend. They all passed. I integrated them in a local shell. It looked fine. But in production, the order of scripts loading caused a race condition. The header micro-frontend depended on a global window.api that the products micro-frontend set up. But the header loaded first. It crashed. I bought into end-to-end tests with Cypress. They ran for twenty minutes. They flaked because of network conditions. After a week, nobody trusted them. I started writing integration tests that mocked the other micro-frontends. But then I was testing my code with fake mocks, missing real bugs.

Performance budgeting became a nightmare. A monolith has one bundle, maybe with lazy loading. You know exactly how much JavaScript you push to the client. With micro-frontends, each team optimizes their own piece. But the browser loads them all. One team adds a heavy charting library. Another pulls in a datepicker with locales. Together, they add up to 3 MB of JavaScript. The user's phone chokes. I had to create a shared performance budget per page, but that required cross-team agreement and a shared monitoring tool. The shiny independent teams concept melts into a coordinated effort again.

Deployment independence is the biggest lie. I deployed my cart micro-frontend with a new API endpoint. It required a new field in the order object. The order service expected that field. But the order service was in a separate micro-frontend that hadn't been updated. For two hours, every order failed. The rollback took fifteen minutes. I learned to always run compatibility tests before deploying. But those tests required the exact same versions of all micro-frontends running together. That meant writing a CI pipeline that pulled the latest from each team, built them together, ran integration tests, and only then allowed a deploy. This pipeline was now slower than the old monolith's deploy.

Feature flags helped. I could keep backward compatibility by supporting both old and new API shapes. But that code accumulates. One year later, I had a micro-frontend that handled three different versions of the product endpoint. It was as complex as the old monolith's conditional logic.

I found that successful micro-frontend architectures only work if you enforce strong contracts. You need a shared design system, a versioned component registry. Every component is published as a package with strict versioning. Every event payload must be defined in a shared schema, validated at runtime. You need automated tests that simulate the entire page with real micro-frontends, not mocks. You need real-time dependency conflict detection in your CI. I used a tool that scanned the lock files of all micro-frontends and reported if two of them used incompatible versions of the same library.

I started writing integration tests that spun up the actual shell and all micro-frontends using Docker. It took eight minutes to run. Every developer had to run it before merging. Initially they complained. Then they realized that the alternative was debugging production outages on Friday nights.

Here is the code for a simple integration test setup I used:

// integration/checkout.spec.js
const { spawn } = require('child_process');
const waitOn = require('wait-on');

describe('Checkout integration', () => {
  let server, shell, header, products, cart;

  beforeAll(async () => {
    // Start all micro-frontends
    header = spawn('node', ['header/server.js']);
    products = spawn('node', ['products/server.js']);
    cart = spawn('node', ['cart/server.js']);
    shell = spawn('node', ['shell/server.js']);

    await waitOn({ resources: [
      'http://localhost:3000', // shell
      'http://localhost:3001', // header
      'http://localhost:3002', // products
      'http://localhost:3003', // cart
    ]});
  });

  it('adds product to cart and shows updated count', async () => {
    await page.goto('http://localhost:3000/products/123');
    await page.click('[data-testid="add-to-cart"]');
    await page.goto('http://localhost:3000/cart');
    const cartCount = await page.textContent('[data-testid="cart-count"]');
    expect(cartCount).toBe('1');
  });

  afterAll(() => {
    header.kill();
    products.kill();
    cart.kill();
    shell.kill();
  });
});

I also learned to monitor the boundaries. I added custom metrics in each micro-frontend for every event dispatch and every API call to another service. I logged the payload shape. When a field went missing, I saw the error immediately in Grafana instead of waiting for user complaints.

// Custom event monitoring
const originalDispatch = window.dispatchEvent;
window.dispatchEvent = function(event) {
  // Send to monitoring
  if (event.type.startsWith('micro:')) {
    fetch('/monitor/event', {
      method: 'POST',
      body: JSON.stringify({ type: event.type, detail: event.detail })
    });
  }
  return originalDispatch.call(this, event);
};

In the end, I don't think micro-frontends are a bad idea. But they are not a shortcut to simplicity. They trade one kind of complexity for another, more distributed kind. If you have a team of five people working on a simple dashboard, you do not need micro-frontends. They will only hurt. But if you have five teams of ten, each owning a distinct business domain, and you cannot coordinate releases, then micro-frontends can help. But only if you are willing to invest heavily in contracts, integration testing, shared components, runtime validation, and performance monitoring.

I have seen teams succeed with micro-frontends. They all had strict shared libraries, a unified build pipeline for integration, and a culture of communication. They did not treat independence as an excuse to ignore the other teams. They treated it as a way to scale ownership while agreeing on the hard rules upfront.

My advice is simple. Start with a monolith. When you feel the pain of coordination, do not instantly jump to micro-frontends. First, try modularizing your code inside a single repo. Use package boundaries. Then, if you still have scaling issues, consider micro-frontends. But be ready for the hidden complexities I described. They will test your patience, your discipline, and your team's ability to work together in ways you never expected.

I still use micro-frontends today. But I do not romanticize them. I know the cost: every time I add a new micro-frontend, I add a new interface that can break. I prepare for that breakage. I write the code that catches it before users see it. And I remind myself that the monolith I broke apart was not the enemy. It was the innocent victim of my desire for independence.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

WebAssembly in Production: Memory, Threading, and Performance Techniques That Actually Work

Nithin Bharadwaj — Tue, 05 May 2026 13:35:24 +0000

I remember the first time I tried to run heavy image processing in a browser. The JavaScript loop stuttered, the page froze, and the user had already closed the tab. That’s when I discovered WebAssembly – a way to run near-native code right inside the browser. But loading a .wasm file is only the beginning. To really speed things up, you need to handle memory, threading, and error cases. Over the years I’ve built a handful of techniques that turn WebAssembly from a fancy demo into a reliable part of any web application. Let me walk you through them, one by one.

First, you have to manage the module’s entire life cycle. If your page loads five different WASM components, you don’t want to fetch and compile them every time the user navigates. I built a simple manager that caches compiled modules and tracks their memory usage. It checks if WebAssembly is supported and falls back to a pure JavaScript version if not. The code below also sets a timeout for slow networks – because nobody wants a spinning loader forever.

class WasmModuleManager {
  constructor() {
    this.cache = new Map();
    this.instances = new Map();
    this.ready = false;
    this.fallbackMode = false;
  }

  async loadModule(url, imports = {}) {
    if (this.cache.has(url)) {
      return this.cache.get(url);
    }

    if (!WebAssembly || !WebAssembly.instantiateStreaming) {
      console.warn('WebAssembly not supported, falling back to JS fallback');
      this.fallbackMode = true;
      return null;
    }

    try {
      const fetchPromise = fetch(url);
      const instantiatePromise = WebAssembly.instantiateStreaming(
        fetchPromise,
        this.buildImports(imports)
      );

      const timeoutPromise = new Promise((_, reject) =>
        setTimeout(() => reject(new Error('WASM load timeout')), 30000)
      );

      const result = await Promise.race([instantiatePromise, timeoutPromise]);
      const { instance, module } = result;

      this.cache.set(url, { instance, module });
      this.instances.set(url, instance);

      this.initMemoryMonitoring(instance);
      this.ready = true;

      console.log(`WASM module loaded: ${url} (${instance.exports.memory.buffer.byteLength} bytes heap)`);
      return instance;
    } catch (error) {
      console.error('WASM load failed:', error);
      return null;
    }
  }

  buildImports(userImports) {
    return {
      env: {
        abort: (msg, file, line, column) => {
          console.error(`WASM abort: ${msg} at ${file}:${line}:${column}`);
        },
        log: (ptr, len) => {
          const bytes = new Uint8Array(this.instance.exports.memory.buffer, ptr, len);
          const msg = new TextDecoder().decode(bytes);
          console.log('[WASM]', msg);
        },
        ...userImports.env,
      },
      ...userImports,
    };
  }

  initMemoryMonitoring(instance) {
    const memory = instance.exports.memory;
    let lastSize = memory.buffer.byteLength;

    setInterval(() => {
      const currentSize = memory.buffer.byteLength;
      if (currentSize > lastSize) {
        console.log(`WASM memory grew: ${lastSize} -> ${currentSize} bytes`);
        lastSize = currentSize;
      }
    }, 2000);
  }

  getInstance(url) {
    return this.instances.get(url);
  }

  dispose(url) {
    const instance = this.instances.get(url);
    if (instance) {
      instance.exports = null;
      this.instances.delete(url);
      this.cache.delete(url);
    }
  }
}

const wasmManager = new WasmModuleManager();
const instance = await wasmManager.loadModule('/compute.wasm');

Now, the biggest mistake people make is copying data back and forth between JavaScript and WASM memory. I used to create new arrays every time I wanted to pass a large buffer. That killed all the speed gains. Instead, you can create a typed array that points directly into the WASM linear memory. No copies. No garbage collector thrashing. The bridge below handles strings, arrays of numbers, and automatically frees allocated memory when you’re done.

class WasmDataBridge {
  constructor(instance) {
    this.instance = instance;
    this.memory = instance.exports.memory;
    this.encoder = new TextEncoder();
    this.decoder = new TextDecoder();
    this.allocated = new Set();
  }

  malloc(size) {
    const ptr = this.instance.exports.malloc(size);
    this.allocated.add(ptr);
    return ptr;
  }

  free(ptr) {
    if (this.allocated.has(ptr)) {
      this.instance.exports.free(ptr);
      this.allocated.delete(ptr);
    }
  }

  writeFloat64Array(arr) {
    const byteLength = arr.length * 8;
    const ptr = this.malloc(byteLength);
    const view = new Float64Array(this.memory.buffer, ptr, arr.length);
    view.set(arr);
    return { ptr, length: arr.length };
  }

  readFloat64Array(ptr, length) {
    const view = new Float64Array(this.memory.buffer, ptr, length);
    return Array.from(view);
  }

  writeString(str) {
    const bytes = this.encoder.encode(str);
    const ptr = this.malloc(bytes.length + 1);
    const view = new Uint8Array(this.memory.buffer, ptr, bytes.length + 1);
    view.set(bytes);
    view[bytes.length] = 0;
    return ptr;
  }

  readString(ptr) {
    const bytes = [];
    const view = new Uint8Array(this.memory.buffer);
    let offset = ptr;
    while (view[offset] !== 0) {
      bytes.push(view[offset]);
      offset++;
    }
    return this.decoder.decode(new Uint8Array(bytes));
  }

  dispose() {
    for (const ptr of this.allocated) {
      this.instance.exports.free(ptr);
    }
    this.allocated.clear();
  }
}

Some WASM functions run for seconds – matrix multiplication, physics simulations, video encoding. If you call them on the main thread, your whole page freezes. I learned to push those computations into a Web Worker. The worker loads the same WASM module and communicates with the main thread using postMessage. With SharedArrayBuffer and cross-origin isolation, you can even share the same memory buffer without copying. Here is a worker pool that reuses workers and handles multiple requests.

// wasm-worker.js
self.addEventListener('message', async (event) => {
  const { id, url, method, args, buffers } = event.data;

  try {
    const { instance } = await WebAssembly.instantiateStreaming(fetch(url));
    const bridge = new WasmDataBridge(instance);

    const wasmArgs = args.map(arg => {
      if (typeof arg === 'string') return bridge.writeString(arg);
      if (Array.isArray(arg)) return bridge.writeFloat64Array(arg);
      return arg;
    });

    const result = instance.exports[method](...wasmArgs);

    let response;
    if (typeof result === 'number' && result !== 0) {
      response = bridge.readFloat64Array(result, 1);
      bridge.free(result);
    } else {
      response = result;
    }

    wasmArgs.forEach((arg, i) => {
      if (typeof arg === 'object' && arg.ptr) {
        bridge.free(arg.ptr);
      }
    });

    self.postMessage({ id, result: response });
  } catch (error) {
    self.postMessage({ id, error: error.message });
  }
});

// main.js
class WasmWorkerPool {
  constructor(workerUrl, maxWorkers = navigator.hardwareConcurrency || 4) {
    this.workers = [];
    this.queue = [];
    this.pending = new Map();
    this.idCounter = 0;

    for (let i = 0; i < maxWorkers; i++) {
      const worker = new Worker(workerUrl);
      worker.onmessage = (e) => this.handleResult(e.data);
      this.workers.push({ worker, busy: false });
    }
  }

  call(method, args = [], buffers = []) {
    return new Promise((resolve, reject) => {
      const id = this.idCounter++;
      this.pending.set(id, { resolve, reject });
      this.queue.push({ id, method, args, buffers, url: this.moduleUrl });
      this.processQueue();
    });
  }

  processQueue() {
    while (this.queue.length > 0) {
      const workerEntry = this.workers.find(w => !w.busy);
      if (!workerEntry) break;

      const task = this.queue.shift();
      workerEntry.busy = true;
      workerEntry.worker.postMessage(task);
    }
  }

  handleResult(data) {
    const { id, result, error } = data;
    const pending = this.pending.get(id);
    if (pending) {
      this.pending.delete(id);
      if (error) pending.reject(new Error(error));
      else pending.resolve(result);
    }

    const workerEntry = this.workers.find(w => w.worker.id === data._workerId);
    if (workerEntry) {
      workerEntry.busy = false;
      this.processQueue();
    }
  }

  terminate() {
    this.workers.forEach(entry => entry.worker.terminate());
    this.workers = [];
    this.queue = [];
    this.pending.clear();
  }
}

WASM doesn’t have garbage collection. Every byte you allocate has to be freed. For many small allocations, like building a mesh or processing a frame, the standard malloc can fragment your heap. I switched to an arena allocator inside the WASM module. You reserve a big block of memory and hand out pointers sequentially. When the frame ends, you reset the offset – instant cleanup.

// arena.c – compile with emcc -O3 -s STANDALONE_WASM -o arena.wasm
#include <stdint.h>

static uint8_t arena[1024 * 1024]; // 1 MB
static uint32_t arena_offset = 0;

void* arena_alloc(uint32_t size) {
    uint32_t aligned = (size + 7) & ~7;
    if (arena_offset + aligned > sizeof(arena)) return 0;
    void* ptr = &arena[arena_offset];
    arena_offset += aligned;
    return ptr;
}

void arena_reset() {
    arena_offset = 0;
}

From JavaScript you call arena_reset() after each frame or batch of work. No memory leaks, no fragmentation.

When you pass a pointer from WASM back to JavaScript, one wrong number can corrupt your whole page. I always validate pointers before reading. The function below checks if the export exists, if arguments are sensible, and if the returned pointer falls inside the memory buffer. It saved me countless hours of debugging.

function safeCall(instance, methodName, ...args) {
  const fn = instance.exports[methodName];
  if (typeof fn !== 'function') {
    throw new Error(`WASM export "${methodName}" not found`);
  }

  args.forEach((arg, i) => {
    if (typeof arg === 'number' && (arg < 0 || arg > 1e9)) {
      throw new Error(`Argument ${i} out of valid range: ${arg}`);
    }
  });

  const result = fn(...args);

  if (typeof result === 'number' && result > 0) {
    const memory = instance.exports.memory;
    if (result >= memory.buffer.byteLength) {
      throw new Error(`WASM returned invalid pointer: ${result}`);
    }
  }

  return result;
}

Not every browser supports WebAssembly. And even if it does, the module might fail to load because of a Content Security Policy or a network error. I keep a pure JavaScript version of every WASM function. A simple feature check at startup decides which one to use. The FastMath class below shows a matrix multiply that falls back when WASM isn’t available.

class FastMath {
  constructor() {
    this.useWasm = false;
    this.instance = null;
  }

  async init() {
    try {
      this.instance = await wasmManager.loadModule('/math.wasm');
      this.useWasm = true;
    } catch {
      console.log('Using JavaScript fallback for math operations');
    }
  }

  matrixMultiply(A, B) {
    if (this.useWasm && this.instance) {
      const bridge = new WasmDataBridge(this.instance);
      const aPtr = bridge.writeFloat64Array(A);
      const bPtr = bridge.writeFloat64Array(B);
      const resultPtr = this.instance.exports.matrixMultiply(
        aPtr.ptr, bPtr.ptr, A.length, B.length / A.length
      );
      const result = bridge.readFloat64Array(resultPtr, A.length * (B.length / A.length));
      bridge.free(aPtr.ptr);
      bridge.free(bPtr.ptr);
      bridge.free(resultPtr);
      return result;
    }

    // JavaScript fallback
    const rowsA = A.length;
    const colsA = A[0] ? A[0].length : 1;
    const colsB = B[0] ? B[0].length : 1;
    const result = new Array(rowsA * colsB).fill(0);
    for (let i = 0; i < rowsA; i++) {
      for (let j = 0; j < colsB; j++) {
        for (let k = 0; k < colsA; k++) {
          result[i * colsB + j] += A[i * colsA + k] * B[k * colsB + j];
        }
      }
    }
    return result;
  }
}

Large WASM modules can take seconds to compile. Streaming compilation lets the browser start compiling while the file is still downloading. I show a progress bar based on the Content-Length header. The code below reads the response as a stream, fires a progress callback, and then compiles the full buffer.

async function loadWasmWithProgress(url, onProgress) {
  const response = await fetch(url);
  const contentLength = response.headers.get('Content-Length');
  const total = parseInt(contentLength, 10);

  const reader = response.body.getReader();
  const chunks = [];
  let received = 0;

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    chunks.push(value);
    received += value.length;
    if (onProgress) onProgress(received / total);
  }

  const blob = new Blob(chunks);
  const arrayBuffer = await blob.arrayBuffer();

  const module = await WebAssembly.compile(arrayBuffer);
  const instance = await WebAssembly.instantiate(module, {});
  return instance;
}

How do you know if the WASM version is actually faster? I profile every call with performance.now(). If a single call takes more than a millisecond, I log it. Over many calls I compute averages. The profiler below tracks totals and counts, so you can compare different implementations.

class WasmProfiler {
  constructor() {
    this.totals = new Map();
    this.counts = new Map();
  }

  call(name, fn, ...args) {
    const start = performance.now();
    const result = fn(...args);
    const elapsed = performance.now() - start;

    this.totals.set(name, (this.totals.get(name) || 0) + elapsed);
    this.counts.set(name, (this.counts.get(name) || 0) + 1);

    return result;
  }

  report() {
    for (const [name, total] of this.totals) {
      const avg = total / this.counts.get(name);
      console.log(`${name}: avg ${avg.toFixed(3)}ms, calls ${this.counts.get(name)}`);
    }
  }
}

With SharedArrayBuffer you can avoid copies between workers entirely. Both the main thread and the worker see the same memory. This is perfect for real-time audio processing or physics. But you need cross-origin isolation headers (COOP and COEP). Once they’re set, you pass the buffer to the worker and both sides can write and read directly. A simple pattern:

// main.js
const sab = new SharedArrayBuffer(1024 * 1024); // 1 MB shared
const worker = new Worker('worker.js');
worker.postMessage(sab);

// worker.js
self.onmessage = (e) => {
  const sab = e.data;
  const view = new Float64Array(sab);
  // read/write directly
};

Finally, I keep the WASM layer clean. JavaScript handles DOM, events, and user interaction. WASM only does number crunching. I create a facade object that the rest of my application talks to. It loads the WASM module once, translates high-level calls into low-level pointer operations, and returns results as plain JavaScript objects.

class WasmFacade {
  constructor(moduleUrl) {
    this.moduleUrl = moduleUrl;
    this.instance = null;
    this.ready = false;
  }

  async init() {
    this.instance = await wasmManager.loadModule(this.moduleUrl);
    this.ready = true;
  }

  async processImage(pixels, width, height) {
    if (!this.ready) await this.init();

    const bridge = new WasmDataBridge(this.instance);
    const pixelPtr = bridge.malloc(pixels.length);
    const pixelView = new Uint8Array(this.instance.exports.memory.buffer, pixelPtr, pixels.length);
    pixelView.set(pixels);

    const resultPtr = this.instance.exports.applyFilter(pixelPtr, width, height);
    const resultLength = width * height * 4;
    const resultView = new Uint8Array(this.instance.exports.memory.buffer, resultPtr, resultLength);
    const result = new Uint8Array(resultView);

    bridge.free(pixelPtr);
    return result;
  }
}

I use this pattern in all my side projects. It keeps the code readable and the performance predictable. Each technique here solved a real problem I faced – slow loads, crashes, frozen UI, memory leaks. Combine them, and you get a WebAssembly integration that feels like part of the platform, not a fragile add-on.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

8 AST Manipulation Techniques to Transform JavaScript Code Automatically

Nithin Bharadwaj — Tue, 05 May 2026 13:24:07 +0000

I remember the first time I tried to change thousands of JavaScript files by hand. I opened each file, found the patterns, and replaced them one by one. It took days. That’s when I discovered AST manipulation – a way to treat code as data and transform it automatically. Let me walk you through eight techniques I use regularly. I’ll keep everything simple, with lots of code you can copy and tweak.

Parsing JavaScript into an AST

Every transformation starts with parsing. You take a string of code and turn it into a tree structure called an Abstract Syntax Tree (AST). Each node in the tree represents a piece of code – a variable declaration, a function call, a literal value. I use @babel/parser because it supports modern syntax like JSX, TypeScript, and optional chaining.

Here’s how I parse any JavaScript source:

const parser = require('@babel/parser');

function parseCode(source, options = {}) {
  try {
    return parser.parse(source, {
      sourceType: 'unambiguous',
      plugins: [
        'jsx',
        'typescript',
        'decorators-legacy',
        'classProperties',
        'optionalChaining',
        'nullishCoalescingOperator',
        ...(options.plugins || [])
      ]
    });
  } catch (error) {
    console.error(`Parse error at line ${error.loc.line}:${error.loc.column}`);
    throw error;
  }
}

If the code has a syntax error, the parser throws. I catch that error and print the exact line and column so I know where to fix things. That’s your foundation. Without a good parse, nothing else works.

Walking the Tree (Traversal)

Once you have an AST, you need to move through it and visit nodes. Think of it like walking through a family tree. You start at the root (the whole program) and go down into children. For each node, you can run a function – we call these “visitors.”

I wrote a simple depth‑first traversal that keeps track of the parent node. That parent reference is gold: it lets you replace or remove a node later.

function traverse(ast, visitors) {
  const stack = [{ node: ast, parent: null, key: null }];

  while (stack.length > 0) {
    const { node, parent, key } = stack.pop();
    const path = { node, parent, key, replaceWith, remove };

    if (visitors.enter) visitors.enter(path);
    const specificVisitor = visitors[node.type];
    if (specificVisitor) specificVisitor(path);

    for (const prop of Object.keys(node)) {
      if (prop === 'type' || prop === 'start' || prop === 'end') continue;
      const value = node[prop];
      if (Array.isArray(value)) {
        value.forEach((child, index) => {
          if (typeof child === 'object' && child !== null) {
            stack.push({ node: child, parent: node, key: { prop, index } });
          }
        });
      } else if (typeof value === 'object' && value !== null) {
        stack.push({ node: value, parent: node, key: prop });
      }
    }

    if (visitors.exit) visitors.exit(path);
  }

  function replaceWith(newNode) {
    const { prop, index } = this.key;
    if (index !== undefined) {
      this.parent[prop][index] = newNode;
    } else {
      this.parent[prop] = newNode;
    }
  }

  function remove() {
    const { prop, index } = this.key;
    if (index !== undefined) {
      this.parent[prop].splice(index, 1);
    } else {
      this.parent[prop] = null;
    }
  }
}

This is the engine that powers every transformation. You write visitors like FunctionDeclaration(path) { ... } and the traversal calls your code for every function in the file.

Writing a Codemod: CommonJS to ES Modules

One of the most useful codemods I ever built converts require calls into import statements. You know the drill – your old codebase uses const foo = require('bar') and you want import foo from 'bar'. Let’s automate it.

I parse the file, scan for CallExpression nodes where the callee is require with a string argument. Then I check the parent: if it’s a variable declarator, I can generate the import.

function requireToImport(code) {
  const ast = parseCode(code);
  const imports = [];
  let output = code;

  traverse(ast, {
    CallExpression(path) {
      if (path.node.callee.name === 'require' && 
          path.node.arguments.length === 1 &&
          t.isStringLiteral(path.node.arguments[0])) {
        const modulePath = path.node.arguments[0].value;
        const parent = path.parent;
        if (t.isVariableDeclarator(parent) && 
            parent.init === path.node) {
          const id = parent.id;
          let importStr;
          if (t.isObjectPattern(id)) {
            const specifiers = id.properties.map(prop => 
              `${prop.value.name} as ${prop.key.name}`
            ).join(', ');
            importStr = `import { ${specifiers} } from '${modulePath}';`;
          } else if (t.isIdentifier(id)) {
            importStr = `import ${id.name} from '${modulePath}';`;
          }
          imports.push({ importStr, start: path.node.start, end: path.node.end });
        }
      }
    }
  });

  // Replace from bottom to top to preserve positions
  imports.sort((a, b) => b.start - a.start);
  imports.forEach(({ importStr, start, end }) => {
    output = output.slice(0, start) + importStr + output.slice(end);
  });

  return output;
}

I sort replacements in reverse order so earlier text positions don’t shift when I replace later ones. That’s a trick I learned the hard way.

Custom Linting Rules

Sometimes ESLint rules don’t cover what you need. Maybe you want to forbid foo && foo.bar and force people to write foo?.bar. You can build a tiny linter that walks the AST and checks patterns.

I write a function that returns an array of violations, each with a message and a source location. The visitor looks for LogicalExpression nodes where the operator is && and the right side is a member expression on the same object.

function lintOptionalChaining(source) {
  const ast = parseCode(source);
  const violations = [];

  traverse(ast, {
    LogicalExpression(path) {
      const node = path.node;
      if (node.operator === '&&' && 
          t.isMemberExpression(node.right) &&
          t.isIdentifier(node.right.object) &&
          node.right.object.name === node.left.name) {
        violations.push({
          message: 'Use optional chaining instead of logical AND guard',
          line: node.loc.start.line,
          column: node.loc.start.column,
          fix: () => {
            const memberExpr = node.right;
            memberExpr.optional = true;
            memberExpr.object = node.left;
            return memberExpr;
          }
        });
      }
    }
  });

  return violations;
}

You can plug these violations into any reporting system. I often print them to the console with the file name and line number. The fix function returns a replacement node – that’s how you can automatically correct the pattern.

Transforming JSX into Function Calls

If you’re building a framework that doesn’t use React’s runtime, you might need to convert <div className="foo">Hello</div> into createElement('div', { className: 'foo' }, 'Hello'). I wrote a transformer that walks JSX elements and builds the equivalent call expression.

The trick is handling attributes, children, and spread attributes. Each attribute becomes a property in an object, and children become extra arguments.

function jsxToCreateElement(ast) {
  traverse(ast, {
    JSXElement(path) {
      const node = path.node;
      const tagName = t.isJSXIdentifier(node.openingElement.name) 
        ? node.openingElement.name.name 
        : t.stringLiteral(node.openingElement.name.name);

      const props = [];
      if (node.openingElement.attributes.length > 0) {
        const propObject = {};
        node.openingElement.attributes.forEach(attr => {
          if (t.isJSXAttribute(attr)) {
            const key = attr.name.name;
            let value;
            if (attr.value === null) {
              value = t.booleanLiteral(true);
            } else if (t.isJSXExpressionContainer(attr.value)) {
              value = attr.value.expression;
            } else {
              value = attr.value;
            }
            propObject[key] = value;
          } else if (t.isJSXSpreadAttribute(attr)) {
            props.push(t.spreadElement(attr.argument));
          }
        });
        props.push(t.objectExpression(
          Object.entries(propObject).map(([k, v]) => 
            t.objectProperty(t.identifier(k), v)
          )
        ));
      } else {
        props.push(t.nullLiteral());
      }

      const children = node.children
        .filter(child => !t.isJSXText(child) || child.value.trim())
        .map(child => {
          if (t.isJSXText(child)) return t.stringLiteral(child.value.trim());
          if (t.isJSXExpressionContainer(child)) return child.expression;
          return child;
        });

      path.replaceWith(
        t.callExpression(
          t.identifier('createElement'),
          [tagName, ...props, ...children]
        )
      );
    }
  });
}

After this, you can run the AST through a generator and get plain JavaScript with no JSX syntax.

Building a Simple Minifier

A minifier shrinks code by shortening variable names, removing whitespace, and eliminating dead code. I built one that collects all identifier names in a scope and renames them to single letters. The AST gives me perfect scope information – I find all VariableDeclarator and FunctionDeclaration nodes, grab their bound names, and assign short aliases.

function minify(ast) {
  const scopes = new Map();
  traverse(ast, {
    FunctionDeclaration(path) {
      const scope = [];
      path.node.params.forEach(param => {
        if (t.isIdentifier(param)) scope.push(param.name);
      });
      scopes.set(path.node, scope);
    },
    VariableDeclaration(path) {
      path.node.declarations.forEach(decl => {
        if (t.isIdentifier(decl.id)) {
          const parentScope = findParentScope(path);
          parentScope.push(decl.id.name);
        }
      });
    }
  });

  let counter = 0;
  const renameMap = new Map();
  const shortNames = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'.split('');

  scopes.forEach((vars) => {
    vars.forEach(name => {
      if (!renameMap.has(name)) {
        renameMap.set(name, shortNames[counter % shortNames.length] + 
          (counter >= shortNames.length ? Math.floor(counter / shortNames.length) : ''));
        counter++;
      }
    });
  });

  traverse(ast, {
    Identifier(path) {
      if (renameMap.has(path.node.name) && !path.isReferencedIdentifier()) {
        path.node.name = renameMap.get(path.node.name);
      }
    }
  });

  const { code: generated } = require('@babel/generator').default(ast, {
    retainLines: false,
    compact: true,
    comments: false,
    minified: true
  });

  return generated;
}

This is just the beginning – you can also fold constants, remove console.log statements, and inline simple expressions.

Type Checking with AST

You don’t always need a full TypeScript compiler. Sometimes you just want to check that variables are used with the right type. I build a lightweight type checker that reads type annotations (like : string) and infers types from literals. Then it compares them when assignments happen.

function typeCheck(ast) {
  const types = new Map();

  traverse(ast, {
    VariableDeclarator(path) {
      if (path.node.id.typeAnnotation) {
        const type = path.node.id.typeAnnotation.typeAnnotation;
        types.set(path.node.id.name, resolveTypeAnnotation(type));
      }
    },
    FunctionDeclaration(path) {
      if (path.node.returnType) {
        const returnType = resolveTypeAnnotation(path.node.returnType);
        types.set(path.node.id.name + ':return', returnType);
      }
      path.node.params.forEach((param, i) => {
        if (param.typeAnnotation) {
          const paramType = resolveTypeAnnotation(param.typeAnnotation.typeAnnotation);
          types.set(param.name, paramType);
        }
      });
    }
  });

  const errors = [];

  traverse(ast, {
    AssignmentExpression(path) {
      const left = path.node.left;
      if (t.isIdentifier(left) && types.has(left.name)) {
        const expectedType = types.get(left.name);
        const actualType = inferType(path.node.right, types);
        if (!isTypeCompatible(actualType, expectedType)) {
          errors.push({
            message: `Type mismatch: expected ${expectedType}, got ${actualType}`,
            line: path.node.loc.start.line
          });
        }
      }
    }
  });

  return errors;

  function resolveTypeAnnotation(node) {
    if (t.isTSStringKeyword(node)) return 'string';
    if (t.isTSNumberKeyword(node)) return 'number';
    if (t.isTSBooleanKeyword(node)) return 'boolean';
    if (t.isTSArrayType(node)) return `Array<${resolveTypeAnnotation(node.elementType)}>`;
    return 'any';
  }

  function inferType(node) {
    if (t.isStringLiteral(node)) return 'string';
    if (t.isNumericLiteral(node)) return 'number';
    if (t.isBooleanLiteral(node)) return 'boolean';
    if (t.isIdentifier(node) && types.has(node.name)) return types.get(node.name);
    return 'any';
  }

  function isTypeCompatible(actual, expected) {
    if (expected === 'any' || actual === 'any') return true;
    return actual === expected;
  }
}

This catches silly mistakes before you ever run the code. You can extend it to handle unions, arrays, and function signatures.

Creating a DSL Compiler

The most fun technique is building a tiny domain‑specific language compiler. You write your own syntax, parse it into a custom AST, then generate JavaScript. I made a simple “math DSL” where x = 5 + 3 becomes a JavaScript variable declaration.

function dslCompiler(dslSource) {
  const dslAst = parseDSL(dslSource);
  const jsAst = generateJSAST(dslAst);
  return require('@babel/generator').default(jsAst).code;
}

function parseDSL(source) {
  const lines = source.split('\n').filter(l => l.trim());
  const nodes = [];
  lines.forEach(line => {
    const match = line.match(/^(\w+)\s*=\s*(.+)$/);
    if (match) {
      nodes.push({ type: 'Assignment', name: match[1], expression: match[2] });
    }
  });
  return { type: 'Program', body: nodes };
}

function generateJSAST(dslAst) {
  const statements = [];
  dslAst.body.forEach(node => {
    if (node.type === 'Assignment') {
      statements.push(
        t.variableDeclaration('const', [
          t.variableDeclarator(
            t.identifier(node.name),
            t.stringLiteral(node.expression)
          )
        ])
      );
    }
  });
  return t.program(statements);
}

You can expand this to handle arithmetic, conditionals, and loops. The AST is just a blueprint – you control every detail of the generated output.

Putting It All Together

Each of these techniques builds on the same idea: treat code as data. You parse it, walk the tree, change nodes, and generate new code. I’ve used these in real projects to migrate thousands of files, enforce team conventions, and even create custom build tools.

Start small. Pick one pattern you hate in your codebase and write a visitor to fix it. Then add another. Before you know it, you’ll have a whole suite of transformations that save you hours every week. The AST is your friend – it gives you total control over JavaScript without ever touching a regular expression.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Forem: Nithin Bharadwaj

5 Custom Java Collector Patterns That Replace Complex Stream Reduction Code

Thread‑Safe Collectors for Parallel Streams

Multi‑Level Grouping

Top‑N Collector

Composing Collectors with Teeing

Personal Touches and Practical Advice

Summary of the Five Patterns

101 Books

Our Creations

We are on Medium

8 Web Worker Techniques That Stop JavaScript From Freezing Your Browser

101 Books

Our Creations

We are on Medium

How to Build Custom Chrome DevTools Extensions for Real-Time Debugging

101 Books

Our Creations

We are on Medium

Python Metaprogramming: 8 Powerful Techniques That Power Django, SQLAlchemy, and Pydantic

Remember, the goal is not to use metaprogramming everywhere. The goal is to know when it is the cleanest solution. For most code, plain functions and classes are enough. But when you face a problem that demands flexibility at runtime, these techniques will be your tools.

101 Books

Our Creations

We are on Medium

# Build Real-Time Data Pipelines With Python, Faust, and Apache Kafka: 8 Proven Techniques

101 Books

Our Creations

We are on Medium

Rust Traits Explained: Write Generic Code Once, Pay Zero Performance Cost

That, for me, is the heart of Rust’s trait system: polymorphism without a performance penalty. You can have both flexibility and speed. No hidden tax.

101 Books

Our Creations

We are on Medium

8 WebRTC Data Channel Techniques for Reliable Peer-to-Peer Communication in JavaScript

Building direct browser-to-browser communication is incredibly satisfying. Your users get faster, more private interactions, and you avoid the cost of relaying all data through a server. I hope these techniques give you a solid starting point for your own peer-to-peer applications.

101 Books

Our Creations

We are on Medium

How to Build a High-Performance HTTP Router in Go With a Radix Tree and Composable Middleware

Start with the code above, add your own features, and measure the results. You'll be surprised how far you can go with just a few hundred lines of Go.

101 Books

Our Creations

We are on Medium

8 Python Automation Techniques Every DevOps Engineer Needs to Master

101 Books

Our Creations

We are on Medium

Micro-Frontends Aren't Simple: Hard Lessons From Breaking Up a Monolith

101 Books

Our Creations

We are on Medium

WebAssembly in Production: Memory, Threading, and Performance Techniques That Actually Work

101 Books

Our Creations

We are on Medium

**8 AST Manipulation Techniques to Transform JavaScript Code Automatically**

Parsing JavaScript into an AST

Walking the Tree (Traversal)

Writing a Codemod: CommonJS to ES Modules

Custom Linting Rules

Transforming JSX into Function Calls

Building a Simple Minifier

Type Checking with AST

Creating a DSL Compiler

Putting It All Together

101 Books

Our Creations

We are on Medium

Composing Collectors with `Teeing`

8 AST Manipulation Techniques to Transform JavaScript Code Automatically