Forem: erndob

P2P For Web Devs, Part 1.1: dht-rpc

erndob — Fri, 20 Mar 2026 05:30:56 +0000

Previous articles:

P2P For Web Devs, Prologue: Why should I care?

P2P For Web Devs, Part 1: Networking

dht-rpc

dht-rpc is a Kademlia based DHT implementation.

Quick refresher:

Distributed Hash Table (DHT) stores data across different nodes in some kind of shared key space.
Kademlia defines how we store data and find the exact node that holds the hash key you are looking for. It's one of the ways to implement a DHT.

Let's build our own DHT and see what happens. I might mention some things that might make no sense yet, but as you read this article we will go back to those concepts and it will click.

We have absolutely nothing now. Let's create the very first node.

import DHT from 'dht-rpc'
const node = new DHT()

This single new DHT() statement does A LOT.

It's an instance of EventEmitter that during constructor execution will:

Create a kademlia routing table
Open UDP sockets and start listening.
Starts watching network interfaces, health, and ticking to react to changes and maintain the routing table.
Tracks all the nodes it's seen and keeps track of their liveliness.
Probes the network to find closest nodes to itself.

Now, at this stage this is a bit useless. It does a lot and at the same time nothing, because we are lacking some key parts.

E.g. nodes identify themselves through a key that is a hash of your public ip and port number. dht-rpc does that for you, it automatically asks another node to ping you, this way getting public IP and confirming that the node is accessible.

But you cannot do that because there's no other nodes at the moment.

Also, nodes start as ephemeral by default. Meaning even in the case of it being publicly accessible, other nodes won't add it to their routing table until the node is deemed stable enough. An ephemeral node functions more as a client, it still interacts with the DHT and can query other nodes for information, but it is not discoverable by other nodes.

So our new DHT() is essentially a client that did not connect to anything. It is technically running, but not really functioning.

Let's fix it.

Bootstrap

We need to have some kind of initial node that can be used by others to join the network. dht-rpc offers a super simple solution for it. E.g. if I want to work in localhost on port 10001 I can do this:

const bootstrap = DHT.bootstrapper(10001, '127.0.0.1')

It works exactly like any other DHT node, if you look at the code, it's a basic new DHT() where we identify that this node:

is not ephemeral
is not behind a firewall as others need to be able to access it
we provide the port number and ip address so that the node can identify itself correctly with a hash(ip+port)
we say that port is static

So this bootstrap node is now live, listening for requests on the port we specified and identifies itself with a consistent id.

At this point in time its routing table is empty as there's no other nodes.

First node

Let's create a node1 that uses this bootstrap.

const node1 = new DHT({
  bootstrap: ['localhost:10001']
})

The node1 initializes its state, the routing table. Currently everything is empty. At this moment your own ID is random, instead of hash(ip+port), as you don't know your own ip and you don't have assigned port.
Initializes networking, opens sockets on OS-assigned ports (as we did not explicitly specify them), starts network watchers and ticks to periodically ping known nodes.
Send a UDP message for FIND_NODE to the bootstrap node, asking to find your own id(the randomly generated one).
Once you receive a response, put that response sender into your routing table.

The previously initiated ticks now will start periodically pinging bootstrap because our routing table now has the bootstrap node in it. We need to keep pinging to track their liveliness as nodes can go down any time.

If you look at the bootstrap node - its routing table is still empty. Even though it is communicating with node1, bootstrap node does not add it into the routing table which makes node1 not discoverable by other nodes. That's because node1 starts as ephemeral.

Upgrading from Ephemeral

After around 20 minutes of running, node1 will try to upgrade itself to a non-ephemeral node. It will do it by:

Send "PING_NAT" messages to up to 5 known nodes. In our current state we know only about the single bootstrap node. In our message we indicate our server socket port number.
If we receive pings on the server socket, that means we are not behind a firewall. And if our IP is stable through all of the pings that have been happening, we upgrade from ephemeral into a proper node.
We change our ID from the randomly generated one into the hash(ip+port).
Because our id changed, we re-bootstrap to get the nodes closest to our new id, by doing the FIND_NODE request again. Resulting in the latest routing table.

After this, if you check in routing table of the bootstrap - our node1 is there! Meaning now bootstrap will also track node1 and keep pinging it.

Third node

Let's add a new node, let's not wait for 20 minutes, and just indicate that this node is not ephemeral.

const node2 = new DHT({
  bootstrap: ['localhost:10001'],
  ephemeral: false
})

If you run this, you will see that node2 immediately learns about bootstrap and node1, and they also immediately become aware of node2. Even though our node2 was never explicitly configured to track node1, the data propagated through out entire network where all 3 nodes know about each other.

Let's kill this node2. After some time, you will see that node2 disappears from bootstrap and node1. So all those pings let's the nodes in the network to know when another node went down, to correctly remove it from tracked nodes.

Many nodes

Let's spin up a 100 nodes.

const clients = Array.from(
  { length: 100 },
  () =>
    new DHT({
      bootstrap: ['localhost:10001'],
      ephemeral: false
    })
)

If you look at the bootstrap node, you will notice that its routing table does not contain all 100 nodes. And if you keep spinning up new nodes, the bootstrap tracked nodes will increase but it will increase slower and slower. I got up to 800 nodes on my machine with bootstrap tracking 123 nodes before my poor Macbook Air started overheating. In the last 100 nodes that were added, the bootstrap added only 2 new nodes into its routing table.

Why?

If every node had to keep track of every single node, ping each other, etc, this wouldn't be able to scale. Kademlia defines how we structure our routing table and how we can still find who we are looking for. A few times I mentioned "distance". That's not geographical distance, that's a mathematical distance between the hash keys, the hash(ip+port) we mentioned. And the routing table groups seen nodes based on this distance, where at certain point you stop tracking newly added nodes in specific distance range. If a node gets a request for a hash, it's fine if it doesn't have that hash in its routing table, it just needs to return the closest nodes to the searched id that it does have. This way through a few hops you can get closer and closer to what you are looking for until you find it. This is also the reason why during bootstrapping you do the FIND_NODE request on your own id - you are trying to fill your own routing table by finding nodes that are close to you.

This is a very rough overview, if you want to go deeper there's a great article on it here.

RPC

So we have our distributed hash table running, we have many nodes connecting to each other, identifying themselves with a hash and we are able to search for any hash.

While that is cool, it's not very useful on its own. Good thing is, the library is called dht-rpc and not just dht. So it allows us to make the nodes do whatever we want through RPC.

A "trick" to it, is that the nodes ids and the data or actions we execute, are all on the same key space so exactly same way of finding nodes close to yourself, can be used to find nodes closest to data. Two clients don't need to know anything about the current state of entire DHT, they can just query the DHT on same hash(whatever) and those two clients will together interact with same node that is closest to this hash. As nodes enter and leave the entire DHT, what is the "closest" node changes over time.

Let's modify the nodes script, to handle custom requests:

const nodes = Array.from({ length: 100 }, () => {
  const node = new DHT({
    bootstrap: ['localhost:10001'],
    ephemeral: false
  })

  node.on('request', function (req) {
    const myId = node.id.toString('hex').substring(0, 5)
    req.reply(Buffer.from(`${myId} did something`))
  })

  return node
})

And now the client:

const client = new DHT({ bootstrap: ['localhost:10001'] })
await client.fullyBootstrapped()
const query = client.query({
  target: Buffer.alloc(32),
  command: 0
})
query.on('data', (x) => console.log(x.value?.toString()))
await query.finished()

We remove the ephemeral: false as this is a client to query the DHT, not be part of it. The target is the key we are looking for, so we care about the node closest to the target. command is just custom number that we can access from the nodes, that's how you implement branching logic for different actions.

Now if we run this, we get:

...
46fc3 did something
1d4c6 did something
12ecc did something
0f40c did something
...

We want to find the node closest to the target, but instead we got data from every single node that got traversed. If you run this client a few times, you will get different outputs, because even though you look for the same target, the traversal changes as each run generates a unique starting routing table. And it depends on the network, which node will respond first. While that is not what we need exactly, it's still good to know that we are able to do this, you can inject code in the entire DHT traversal logic, and you can capture responses from that traversal on the client side.

Read from the closest node

To actually know which response is from the closest node, we can add this line:

console.log('closest reply', query.closestReplies[0].value.toString())

Now, if you run this many times, you will get different results from the query as the traversal is different, but the last line will always be the same. It's always the same node that is closest to your target. If you randomize your target, like target: crypto.randomBytes(32), you will start seeing different closest reply between runs.

Writing to closest node

Reading replies when there's nothing written is not very useful. Or if you need to execute any other action just from the closest node and not the entire traversal.

Let's clean up the entire solution and implement the proper logic.

New DHT nodes:

const nodes = Array.from({ length: 100 }, () => {
  const node = new DHT({
    bootstrap: ['localhost:10001'],
    ephemeral: false
  })

  const state = new Map()

  node.on('request', function (req) {
    const target = req.target.toString('hex')
    switch (req.command) {
      case 0: // Read
        req.reply(state.get(target))
        return
      case 1: // Write
        const previousValue = state.get(target)
        state.set(target, req.value)
        req.reply(previousValue)
        return
    }
    return req.error(20) // Our own defined error code
  })

  return node
})

We store the state in memory, as a simple Map where the key is the target. So you can imagine a single node might be closest to multiple targets and store all of them.
We start using that command property, so clients can indicate what they want to do and requests can handle it. Write command also responds with the previous value.
And if it is an unknown command, we return our custom error code instead.

Now, let's implement our client:

const client = new DHT({ bootstrap: ['localhost:10001'] })
await client.fullyBootstrapped()
const target = crypto.createHash('sha256').update('data key').digest()
const q = client.query({ target, command: 0 })
await q.finished()

const res = await client.request(
  { target, command: 1, value: Buffer.from('write me!') },
  q.closestNodes[0]
)
const id = res.from.id.toString('hex')
console.log(`${id} responded with ${res.value?.toString()}`)

First, we create target key as a sha256 hash. Before I did just a manual buffer of 32 bytes, which does work, but we want to have uniformly distributed keys. You want to make sure you are not accidentally hitting collisions and that the keys are distributed well between nodes and not accumulating among just a few of them.

Then we do exactly same client.query like before. This time I do not care about what it is responding, I just want to get the closest node to the target.

After the query, now I can do client.request. The request body is almost same as it was with query, except our command changed to 1 and we added a value. But the most important thing is that request allows us to pass a specific node, so we can use the q.closestNodes[0] from the read query.

Unlike a query that traverses the DHT, request will make just a single call to the specified node. It hits the same callback we configured in our nodes logic.

If you run the client a few times, you will get the same id each time, as it is same node that is closest to target. And you will get undefined as value first time, and write me! the second time.

You can make exactly same write request but with a query, that is completely valid, but you will end up with many nodes doing that same write as the DHT is traversed.

Reliability

We have strong building blocks already, but there's one issue. What happens if the node that holds data disconnects? Or, even worse, nothing bad happens, instead a new node joins and it happens to be closer to the target. In that case subsequent closestReplies[0] requests start showing undefined even when you had data. Your expectations break even if none of the nodes broke.

Good thing, DHT allows us to commit when we do a query.

const q = client.query(
  { target, command: 0, value: Buffer.from('write me!') },
  { commit: true })

The query looks almost identical to before, except we added { commit: true }, and the value that before was on client.request now is in the client.query.

When you indicate that a query should be committed, once the initial pass through travels the DHT triggering all your traversal requests and populating the closestNodes/closestReplies, it automatically executes same request again but only to the 20 closestReplies and this time it passes the token.

The token is a simple hash of client IP and a secret that got returned by the node during response of DHT traversal. If you make a new request to same node including the token you got from previous request, it gets validated that the hash is correct for your IP address. This gives you a super basic and lightweight auth if you want to track and control who can write or access the data(keep in mind IPs are not static and clients can share them, so it's not appropriate for anything serious). Or you can use it as a simple proof of commit, meaning your node can know that it was recognized as one of the "nearest" to the target.

Here's how our new node request handler looks:

node.on('request', function (req) {
  switch (req.command) {
    case 0: // Read or Write
      const target = req.target.toString('hex')
      const previousValue = state.get(target)
      if (req.token) {
        // this is a commit, we write
        state.set(target, req.value)
      }
      req.reply(state.get(target))
      return
  }
  return req.error(20)
})

Compared to before, we combined our separate commands for read and write into a single command. Instead, we know it is a write if a token exists. Token does not need any validation because DHT itself already validated it.

With this, if you run our query, 20 nodes closest to the target will store the value into their state. Now, even if the closest node dies, you still have your data in other nearby nodes.

You now also do not need a separate client.request for the write, everything is done in this one query.

Now, we can also adjust how we read data, instead of looking at q.closestReplies[0] like we did before, we need to find any node that returns data instead.

for await (const res of client.query({ target, command: 0 })) {
  let value = res.value?.toString()
  if (value) {
    console.log(`${res.from.id.toString('hex')} returned ${value}`)
    break
  }
}

Note that in this query, we do not pass commit: true and the value. We simply read the responses as they come in, once any of the nodes returns it - we break early and are done. This way even if a new closest node does not have the value, it does not matter.

If you run the current code a few times, you will notice that between runs you get different node responding with the value but still in the same subset of nodes.

Summary

We did a lot! We got a very powerful building block that allows us to connect arbitrary amount of machines without a special coordinator. And we got a powerful RPC logic where we can make these nodes do whatever we want.

While the examples shown here do work, it is not production code. It has problems like not necessarily getting the latest data, not dealing with malicious nodes, not having encryption, etc. The goal was to show core patterns that give you ideas on what you can do, not how to do production ready DHT.

We will see how some of these problems are solved in the other modules that build on top of dht-rpc.

P2P For Web Devs, Part 1: Networking

erndob — Thu, 12 Mar 2026 03:23:14 +0000

Previous article in this series: P2P For Web Devs, Prologue: Why should I care?

Hyperswarm

When you start with Pear runtime, one of the first things you encounter is Hyperswarm. It describes itself as:

A high-level API for finding and connecting to peers who are interested in a "topic."

Let's dissect this seemingly simple description.

Peers are processes that communicate through a network, with a key distinction being that they do both - make requests and receive requests. You will often see a term "node" too, these are used almost interchangeably in P2P context, as all peers are nodes.

Swarm in Hyperswarm is a group of peers that are all connected to each other. E.g. a group call with 4 people in it, is a swarm of 4 peers.

Finding means we are solving the equivalent of DNS lookup, we need to get IP addresses of the peers to connect to.

Connecting is the step after "finding". When you get an IP address of a server, usually you connect with HTTP request on a predefined port number. Here it is more complex, as who we are connecting to is not conveniently listening on port 443.

Topic is a way for peers to find each other. E.g. in a phone call of 4 people, a topic would be a call id. Someone else could join this call without knowing the other 4 people, they just need the call id.

A key point to understand is that peers are not equal to an IP address. Your phone is a peer. But a phone is a mobile device that is constantly changing IPs as you move. So a topic would match you to a peer, but translating peer to an IP address is a separate issue.

Now that's a lot of information packed into that short sentence describing Hyperswarm.

Let's reduce scope, and go deeper.

Hyperswarm is built on top of Hyperdht.

HyperDHT

The Hyperswarm DHT uses a series of holepunching techniques to make sure connectivity works on most networks, and is mainly used to facilitate finding and connecting to peers using end to end encrypted Noise streams.

We got a layer deeper, we got new terms, and the description is twice as long now.

DHT - Distributed Hash Table. That's a hash table, that instead of living in memory of a single machine, is actually across many different machines. The different machines can hold a range of data from the table, and all of them together form the full table. E.g. Cassandra DB is an implementation of DHT.
In our context, DHT is what we can use to implement the "finding" part, that's where we can have information about existing peers, their IP addresses, topics. It's the metadata layer that is accessed by the entire network, without a central server.

holepunching - that's the connection part between peers. It solves the previously mentioned issue of "not conveniently listening on port 443".

end to end encrypted Noise streams - that's the equivalent of SSL on the web. The noise protocol is a lot more lightweight compared to SSL and does not require a certificate authority. Which makes it the ideal choice for embedded systems and P2P systems that do not want to depend on some central certification issuer.

So in this module we got some terms not only about what is being done, but also on how it is being done.

Another line in the description of hyperdht is:

Built on top of dht-rpc.

dht-rpc

Make RPC calls over a Kademlia based DHT.

Remote IP / firewall detection

Easily add any command to your DHT

Streaming queries and updates

I think we are deep enough! This is a core module that focuses on the DHT itself.

Kademlia based DHT, DHT's can be implemented in different ways. E.g. the more controlled and trusted nodes will usually implement DHT with a hash ring. In the case of P2P network where peers are connecting and disconnecting all the time, are untrusted and the scale can easily reach millions of peers, Kademlia is the standard. These different approaches define how the node with the data you need is actually found.

I mentioned that we use DHT here for the metadata, for resolving the peer to IP maps. But that's just the use case in the hyperswarm module. This dht-rpc module is just a generic implementation, giving you the ability to command and query your DHT for whatever custom data you need.

Now that we have the vocabulary and understand the problem scope, we can start looking at how these problems are actually solved.

In the next articles I will go through implementation details of each of those modules, building up from dht-rpc to Hyperswarm.

Next article: P2P For Web Devs, Part 1.1: dht-rpc

P2P For Web Devs, Prologue: Why should I care?

erndob — Wed, 11 Mar 2026 07:44:34 +0000

What is it?

Peer-to-peer (P2P) is when devices communicate with each other directly, instead of through a server.

Have I used it before?

P2P is heavily used in a lot of highly distributed systems when you want to avoid having a single point of failure with a centralized orchestrator. You likely used some kind of cloud service that at one of the layers is built with P2P.

You have likely used many P2P based services as a consumer, maybe without realizing it:

Windows Updates are distributed through "Delivery Optimization", where they combine regular HTTP servers, cache layers, and P2P distribution. That P2P distribution is absolutely crucial, as Windows machines are downloading the update, they start sharing the parts they downloaded with other machines. In case of machines on the same network, the update can spread through the network without even touching public internet. You can imagine absolutely massive savings in bandwidth that Microsoft gets from this. This also results in increased reliability and faster download speeds for consumers.
A lot of real time communication apps use P2P for video/voice, often with a mixed architecture.
Things like remote desktop applications will often use P2P to connect you directly.
If you ever used a torrent client to download files - that's P2P.

So what? That does not apply to the majority of applications.

You likely noticed a pattern there - it is either static files like updates, or real time communication where you do not care about state.

But what if we took a step back, and tried to build consumer applications with regular data, that is P2P based?

Of course we would have a lot of things to solve. But let's ignore that for now. Let's think about what a P2P application would give us:

Cost. With state and processing on device, and communication P2P, suddenly the entire cost model is different.
Latency. Less intermediary hops, less latency.
Reliability. No server, means no downtime when a cloud provider goes down. Additionally P2P applications are resilient to internet outages for home/office networks.
Security. Data on device instead of a massive server with everyones data in one place removes a huge area of potential data leaks.
Privacy. This is especially vital now, as AI enables surveillance at an unprecedented scale.
Environment. Data centers have a global impact and localized impact to communities that live close to them. P2P uses devices consumers already own.

If it is so great, why is it not more widely used?

It is hard. There are a lot of problems to solve that have been solved on regular stacks already. Certain aspects require compromises.

There's a lack of incentives.
Modern web development is highly run by FAANG. They are responsible for majority of open source tools that became industry default(React), they control the runtime(Chrome), they control languages(C#), they control the servers(AWS), they control how we store our code(GitHub) and now they also control the AI agents that increasingly do the coding. They have zero incentive to develop tooling for applications that remove the need to pay them and give control to the consumer. They will use these technologies internally for reliability, or in their products to save on bandwidth, but they have zero incentives to develop a full platform for P2P applications. WebRTC by Google is the closest thing we have, but its focus is on real time data streaming. It's developed mostly to strengthen Web as a platform, enabling audio/video/screen sharing in-browser, so Google can compete vs native apps. Not to enable general P2P applications.

What about crypto currency and blockchains?

I must mention crypto as that is the most successful P2P experiment we have. It offers a global, trusted database that anyone can interact with and build upon.

While it is successful, these apps navigate complexity that regular apps do not have. Like, consensus mechanisms so everyone agrees on what the trusted state is, complex monetary incentives, paid transactions, paid storage, etc.

If you take a step back from what is possible with blockchains now, say "we don't need half of this", and build something that is for consumers and not finance, you suddenly get a powerful technical stack that can be adapted for regular applications.

It is useful and can be adapted, but no one wants to solve those hard problems on their own.

That's where Pear comes in. It's a P2P stack that you can use to build consumer applications. If you look at the docs, the index page links to 106 javascript modules taking care of different things.

It feels magical, a few cli commands, a few lines of javascript, and you have working demos of P2P applications.

Work feeling like magic because it is easy is great, work feeling like magic because you don't understand what is happening is a time bomb.

The goal of this series is for myself to learn the Pear stack and demystify it from the web developer perspective.

Instead of approaching it from the end - the pear init, we will focus on the core building blocks of pear runtime. Develop understanding and intuition for how it actually runs, so we know what we can build and how to troubleshoot.

In this series I will focus on the P2P Modules, these are core modules that cover networking, persistence, synchronization and storage.

Next article: P2P For Web Devs, Part 1: Networking