Forem: vasa

Under the hood: AvionDB P2P Syncing

vasa — Fri, 22 May 2020 13:58:23 +0000

One of the big challenges while working with distributed databases is to sync/replicate data between the peers.

There are multiple things that make replication challenging in distributed systems:

Access Control: There should be a way to control the access to your data, so that you can decide which peer can access your data with what permissions (read, write, etc.)
Different Runtimes: The replication should work across different devices: your PC, mobile, browser, etc. But all these devices don't have comparable networking, storage & compute resources.
Restrictive Networks: Firewalls, NATs don't allow all the peers to talk to each other. So, we have to come up with new ways to connect peers that live behind these restrictive networks.

In the next couple of posts, we will discuss how we at Dappkit are working to make data replication developer friendly so that you as a developer don't have to think about all these challenges.

This post goes through an example of how today we replicate data between the peers using AvionDB, and our plans to make it much easier than what it is today.

A peer-to-peer to-do list

Well, a to-do list is not an app that will make you go nuts, but it does show the process of data replication with an easy to understand the application.

The app can be divided into 2 parts:

Creating, listing & updating to-do(s).
Syncing the to-do list across different peers.

Let's get right into it. You can find the working code & the final to-do app demo here.

Creating, listing & updating to-do(s)

This is the easy part of the app. We create a simple interface using bootstrap to create, list & update to-do(s).

You can see a QR code too, which we will use for scanning the URL so that you can sync the to-do list on your mobile. You can also use a link that you can directly put in your browser or share it with anyone who wants to sync the to-do list.

Now, we need to use AvionDB to store, list & update the to-do(s) that will be stored locally in your browser.

AvionDB uses IPFS internally to store its data, so we need to import AvionDB & IPFS.

We use the CDN links and add them to the public/index.html

so that we can access them across the whole application.

If you want to use npm modules, here are the modules for AvionDB & IPFS.

Now, we need to initialize AvionDB & IPFS.

This creates a Database named "TodoList", and a Collection named "todos" in it.

Now, we need to add, list & update todo in the "todos" collection.

Add To-do(s)

You can use functions like insert or insertOne to add todo(s).

Note that each record that is added in the collection will have an \_id field added to it automatically, which serves as a unique identifier of the record. You can override the default \_id by adding an \_id field with your record.

You can find more information on the functions here.

Get To-do(s)

You can fetch the to-do(s) using functions like find & findOne.

This query returns all the to-do(s) in the "todos" collection.

You can find more information on the functions here.

Update To-do(s)

You can update the to-do(s) using functions like update, updateMany, findOneAndUpdate, etc.

This updates the isDone status of a to-do.

You can find more information on the functions here.

Now, as we have created a simple todo app to store, fetch & update the to-do(s), let's see how we can sync/replicate our to-do(s) with multiple devices.

Syncing the to-do list across different peers

Now before going into the code, let's first understand a few concepts that will help you understand peer-to-peer syncing.

Let's take a real-world example.

As we are going through COVID pandemic, Ross & Rachel are attending their classes over the online lectures.

Now, Ross & Rachel want to exchange their notes & do a group study to prepare for their test.

In order to do so, here is what to need to do:

Exchange their email addresses.
Find a common platform to use for communication (Skype, Hangouts, Zoom, etc.). Maybe they also need to decide which language (English, French, Chinese, etc.) they would use to discuss over the call.
Send an invite to their fellow participant.
Join the call, exchange notes & do a group study.

This is very similar to what we are going to do to sync data between the AvionDB peers.

In order to sync the data between any 2 (or more) peers, we need to:

First, find each-others addresses by which we can connect.
Find a common protocol (WebSocket, WebRTC) that both peers can use to communicate.
Establish a connection using the addresses & the common protocol.
And at last sync the data (to-do list).

Let's go through each of these points and see how this works under the hood.

Finding Addresses

Each peer in the network has a unique address that can be used to uniquely identify the peer in the network, known as peerIds.

In the case of AvionDB, we use Libp2p (an internal part of IPFS) peerIds, which look something like this: Qme499EjQog7UjvuJiduzw1UKMe6hZ1rZ1wQrxz7qpq7TS.

These peerIds are generated using a project called Multiformats, which is used to create interoperable, future-proof protocols.

You don't need to go too deep into Libp2p or Multiformats. You just need to know that we use the peerIds as addresses (email Id, or a mobile number) to identify & communicate with other peers in the network.

Common Protocol for Communication

As we talked about Different Runtimes at the beginning of this post, different devices have different networking, storage & compute resources.

This means that a browser does not have as much networking, storage & compute resources as a PC.

As we talked about the Ross & Rachel example, we saw that we needed to decide on a common platform & language that they could use to communicate.

Similarly, a browser, a PC and a mobile need to decide what common protocols they use so that they can communicate with each other, despite their differences in networking, storage & compute resources.

A fact that you should know here is that all these platforms support websocket protocol over TCP. So, here websocket is the "common platform or language" that all the devices can use to communicate with each other.

Now, we have the common protocol, but there are restrictive environments such as firewalls & NATs that restrict the peers from discovering each other.

Here is where circuit-relay comes into the picture.

Circuit Relay

In p2p networks there are many cases where two nodes can't talk to each other directly. That may happen because of network topology, i.e. NATs, or execution environments - for example browser nodes can't connect to each other directly because they lack any sort of socket functionality and relying on specialized rendezvous nodes introduces an undesirable centralization point to the network. A circuit-relay is a way to solve this problem - it is a node that allows two other nodes that can't otherwise talk to each other, use a third node, a relay to do so.

How does circuit relay work?

Here is a simple diagram depicting how a typical circuit-relay connection might look:

Peer A tries to connect to Peer B but, UH-OH! There is a firewall in between that's preventing it from happening. If both Peer A and Peer B know about a relay, they can use it to establish the connection.

This is what it looks like, in simplified steps:

Peer A tries to connect to Peer B over one of its known addresses
Connection fails because of firewall/NAT/incompatible transports/etc...
Both Peer A and Peer B know of the same relay - Peer C
Peer A falls back to dialing over Peer C to Peer B using its '/p2p-circuit' address, which involves:
Peer A sends a HOP request to Peer C
Relay extracts the destination address, figures out that a circuit to Peer B is being requested
Relay sends a STOP request to Peer B
Peer B responds with a SUCCESS message
Relay proceed to create a circuit over the two nodes
Peer A and Peer B are now connected over Peer C

That's it!

What's up with this `HOP` and `STOP`?

Circuit relay consists of two logical parts — dialer/listener and relay (HOP). The listener is also known as the STOP node. Each of these — dial, listen, and relay — happen on a different node. If we use the nodes from the above example, it looks something like this:

The dialer knows how to dial a relay (HOP) - Node A
The relay (HOP) knows how to contact a destination node (STOP) and create a circuit - Relay node
The listener (STOP) knows how to process relay requests that come from the relay (HOP) node - Node B

Fun fact - the *HOP* and *STOP* names are also used internally by circuit to identify the network message types.

A few caveats (and features)

There are a couple of caveats and features to be aware of:

A Relay will only work if it already has a connection to the STOP node
No multihop dialing is supported. It's a feature planned for upcoming releases from Libp2p (no date on this one)
multihop dialing is when several relays are used to establish the connection.
It is possible to use explicit relay addresses to connect to a node, or even to listen for connections on. We will talk more about this below.

A word on circuit relay addresses

A circuit relay address is a multiaddress that describes how to either connect to a peer over a relay (or relays), or allow a peer to announce it is reachable over a particular relay or any relay it is already connected to.

Circuit relay addresses are very flexible and can describe many different aspects of how to establish the relayed connection. In its simplest form, it looks something like this:

/p2p-circuit/ipfs/QmPeer

If we want to be specific as to which transport we want to use to establish the relay, we can encode that in the address as well:

/ip4/127.0.0.1/tcp/65000/ipfs/QmRelay/p2p-circuit/ipfs/QmPeer

This tells us that we want to use QmRelay

located at address 127.0.0.1 and port 65000.

/ip4/127.0.0.1/tcp/65000/ipfs/QmRelay/p2p-circuit/ip4/127.0.0.1/tcp/8080/ws/ipfs/QmPeer

We can take it a step further and encode the same information for the destination peer. In this case, we have it located at 127.0.0.1 on port 8080 and using a Web sockets transport!

/ip4/127.0.0.1/tcp/65000/ipfs/QmRelay/p2p-circuit

If a node is configured with this address, it will use the specified host (/ip4/127.0.0.1/tcp/65000/ipfs/QmRelay) as a relay and it will be reachable over this relay.

There could multiple addresses of this sort specified in the config, in which case the node will be reachable over all of them.
This is useful if, for example, the node is behind a firewall but wants to be reachable from the outside over a specific relay.

Other use-cases are also supported by this scheme, e.g. we can have multiple hops (circuit-relay nodes) encoded in the address, something planed for future releases.

In our to-do list example we created a relay peer with the following multiaddress:

/dnsaddr/node1.dappkit.io/tcp/6969/wss/p2p/QmfLwmXF25u1n5pD8yXcbmZew3td66qPU1FroWNrkxS4bt

You can use this relay to connect your peers too!

Now as we have discussed how to get through different runtimes & restrictive networks, let's now move on to see how to establish a connection.

Establish a connection

In order to connect to any peer using IPFS, we have a function ipfs.swarm.connect(multiaddress) where multiaddress is the address of any peer that a peer wants to connect to.

So, for example, if Peer A wants to connect to the relay – Peer C then we will do something like this:

After, we have connected Peer A & Peer B with the relay – Peer C we can now use the /p2p-circuit address of Peer B to connect to Peer A or vice-versa.

Voila! You have now connected the peers 🎉 Now we just need to replicate the data between the 2 peers.

Sync data between the peers

In order to sync data between the peers, we need the database address of "TodoList" database & the name of the collection that we created above.

To get this information we need to communicate this data between the peers. For this, we use something called Publish/Subscribe

Publish/Subscribe is a system where peers congregate around topics they are interested in. Peers interested in a topic are said to be subscribed to that topic:

Peers can send messages to topics. Each message gets delivered to all peers subscribed to the topic:

So, if we create a topic: peerId of Peer A so that anyone who is subscribed to peerId of Peer A can get the messages from Peer A. This way Peer A can publish the database address of "TodoList" database & the name of the collection to all its subscribers in one go!

Here is the example code showing Peer B subscribed to Peer A using its peerId and using the published message (msg) to open the database & the collection.

Here is an example code showing Peer A publishing the message (msg) to the topic peerIdA.

And that's it! This is how we currently sync data between peers in AvionDB.

Here is a working demo of the to-do list app.

Here are some useful links:

Check out the live app here.
Check out the code here.
Check out AvionDB here.

But this is not something we want you to go through when you are creating your application. We want to give developers a simple way to sync data that abstracts away all this complexity of different runtimes, restrictive networks, etc.

So we are working on making this a better experience for the developers so that they can focus on building applications rather than tackling these core issues.

If you want to keep a close track on our progress then feel free to reach us out on our Discord server.

NOTE: We have not talked about Access Control in this post, but we have a ton of examples already built for different Web 2.0 & Web 3.0 based access control. Feel free to check them out here. We will be back with posts on access controller too. Meanwhile, if you have any questions, feel free to reach us out on our Discord server.

AvionDB: A MongoDB-like Distributed Database

vasa — Sat, 09 May 2020 20:00:40 +0000

In the past few months we have been getting this question a lot:

What do I use for managing dynamic data for my Decentralized App?

We have the InterPlanetary File System for storing static data, but what about the data that keeps changing?

There are a number of existing distributed databases like GUN, OrbitDB, Scuttlebutt, Textile Threads, etc.

Although these existing databases are great, we wanted a distributed database that will feel familiar when you are coming from Web 2.0 & still has all the features that you would expect from a distributed database.

Something like a distributed MongoDB.

Enter AvionDB

AvionDB is a distributed database built on top of OrbitDB & IPFS.

When we started our work on AvionDB, we had a few design goals in our mind.

Design Goals

Familiar developer interface: MongoDB/Mongoose like interface.
Local-first: Your app can work locally without the internet. For example, you can take notes on your notepad without (or even sync with other devices) connecting to the Internet.
Privacy-first: All the data, whether it is on your device or on out in the network, it should be encrypted by default.
Cross-platform: The database can be used natively on Mobile, Web & Desktop platforms.

How does it work?

AvionDB uses OrbitDB stores to model MongoDB-like Databases.

Each AvionDB instance can have several Databases.
Each Database can have several Collections.
Each Collection can have several Documents.

The Javascript implementation works both in Browsers and Node.js with support for Linux, OS X, and Windows. The minimum required version of Node.js is now 8.6.0 due to the usage of ... spread syntax. LTS versions (even-numbered versions 8, 10, etc) are preferred.

Using AvionDB with NodeJS

Install

// Using npm
npm install --save aviondb

// Using Github
npm install git+https://github.com/dappkit/aviondb.git

Usage

// Import modules
const AvionDB = require("aviondb");
const IPFS = require("ipfs");
const ipfs = new IPFS();

const runExample = async () => {
  await ipfs.ready;

  // Creates a db named "DatabaseName"
  const aviondb = await AvionDB.init("DatabaseName", ipfs); 

  // Returns the List of database names
  await AvionDB.listDatabases()
  // prints ['DatabaseName']

  // Creates a Collection named "employees"
  const collection = await aviondb.initCollection("employees");

  // Returns the List of collection names
  await aviondb.listCollections() 
  // prints ['employees'] 

  // Adding an employee document
  await collection.insertOne({
    hourly_pay: "$15",
    name: "Elon",
    ssn: "562-48-5384",
    weekly_hours: 100,
  });

  // We also support multi-insert using collection.insert()
  // See https://github.com/dappkit/aviondb/blob/master/API.md


  // Search by a single field Or many!
  var employee = await collection.findOne({
    ssn: "562-48-5384", 
  });

  // We also support find(), findById()
  // See https://github.com/dappkit/aviondb/blob/master/API.md

  // Returns the matching document
  console.log(employee); 
  // Prints the above added JSON document


  // Update a document
  var updatedEmployee = await collection.update(
   { ssn: "562-48-5384" },
   { $set: { hourly_pay: '$100' } }
  );

  // We also support updateMany(), findOneAndUpdate()
  // See https://github.com/dappkit/aviondb/blob/master/API.md

  // Returns the updated document
  console.log(updatedEmployee); 
  // Prints the updated JSON document


  await collection.close(); // Collection will be closed.
  await aviondb.drop(); // Drops the database 
  await aviondb.close(); // Closes all collections and binding database.
  await ipfs.stop();
};

runExample();

Using AvionDB with Browser

through Browserify

Same as in Node.js, you just have to browserify to bundle the code before serving it.

Note: The code uses es6, so you have to use babel to convert the code into es5 before using browserify.

through webpack

Same as in Node.js, you just have to webpack to bundle the the code before serving it.

Note: The code uses es6, so you have to use babel to convert the code into es5 before using webpack.

from CDN

Instead of a local installation (and browserification) you may request a remote copy of AvionDB from unpkg CDN.

To always request the latest version, use the following:

<!-- loading the minified version -->
<script src="https://unpkg.com/aviondb/dist/aviondb.min.js"></script>

CDN-based AvionDB provides the AvionDB constructor as a method of the global window object. Example:

// create an AvionDB instance
const aviondb = await AvionDB.create("DatabaseName", ipfs)

Usage

<html>
  <head>
    <title>AvionDB Example</title>
    <script src="https://cdn.jsdelivr.net/npm/ipfs/dist/index.min.js"></script>
    <script src="https://unpkg.com/aviondb/dist/aviondb.min.js"></script>
  </head>
  <body>
    Open your console logs to see AvionDB in action!
  </body>
  <script type="text/javascript">
  const runExample = async () => {
    const ipfs = await window.Ipfs.create();
    await ipfs.ready;

    // Creates a db named "DatabaseName"
    const aviondb = await AvionDB.init("DatabaseName", ipfs);

    // Returns the List of database names
    await AvionDB.listDatabases()
    // prints ['DatabaseName'] 

    // Creates a Collection named "employees"
    const collection = await aviondb.initCollection("employees");

    // Returns the List of collection names
    await aviondb.listCollections() 
    // prints ['employees'] 

    // Adding an employee document
    await collection.insertOne({
      hourly_pay: "$15",
      name: "Elon",
      ssn: "562-48-5384",
      weekly_hours: 100,
    });

    // We also support multi-insert using collection.insert()
    // See https://github.com/dappkit/aviondb/blob/master/API.md


    // Search by a single field Or many!
    var employee = await collection.findOne({
      ssn: "562-48-5384", 
    });

    // We also support find(), findById()
    // See https://github.com/dappkit/aviondb/blob/master/API.md

    // Returns the matching document
    console.log(employee); 
    // Prints the above added JSON document


    // Update a document
    var updatedEmployee = await collection.update(
    { ssn: "562-48-5384" },
    { $set: { hourly_pay: '$100' } }
    );

    // We also support updateMany(), findOneAndUpdate()
    // See https://github.com/dappkit/aviondb/blob/master/API.md

    // Returns the updated document
    console.log(updatedEmployee); 
    // Prints the updated JSON document


    await collection.close(); // Collection will be closed.
    await aviondb.drop(); // Drops the database 
    await aviondb.close(); // Closes all collections and binding database.
    await ipfs.stop();
  };

  runExample();
  </script>
</html>

Demo Example of Firebase Authentication with AvionDB

Check out the app here.
Check out the code here

Frequently Asked Questions

Are there implementations in other languages?

We are working to implement AvionDB for following languages:

NodeJS & Browser JS
Typescript
Golang
Python

The best place to find out what is out there and what is being actively worked on is likely by asking in the Discord.

If you want or are planning to create an implementation in a language that is not listed here, then feel free to reach us out and discuss about it in the Discord.

Where can I see your Roadmap?

You can find our Roadmap here. The features in the Roadmap are taken from 2 separate issues(#7, #8) which individually maintain a list of feature proposals related to OrbitDB-specific improvements & AvionDB-specific improvements respectively.

The Roadmap is an open discussion, feel free to add your suggestions, comments.

What mongodb features does aviondb support?

You can find all the supported MongoDB-like features in our API docs.

Contributing

Take a look at our organization-wide Contributing Guide.

As far as code goes, we would be happy to accept PRs! If you want to work on something, it'd be good to talk beforehand to make sure nobody else is working on it. You can reach us on Discord, or in the issues section.

If you want to code but don't know where to start, check out the issues labeled "help wanted", "discussion".

Please note that we have a Code of Conduct, and that all activity in the @dappkit organization falls under it. Read it when you get the chance, as being part of this community means that you agree to abide by it. Thanks.

Contact Us

Feel free to reach us out on Discord.

How to Write Upgradable Smart Contracts (Smart Contract Versioning)

vasa — Wed, 26 Feb 2020 21:37:36 +0000

This article was first published on our open-source platform, SimpleAsWater.com. If you are interested in IPFS, Libp2p, Ethereum, Zero-knowledge Proofs, Defi, CryptoEconomics, IPLD, Multi formats, and other Web 3.0 projects, concepts and interactive tutorials, then be sure to check out SimpleAsWater.com.

Smart contracts are immutable, by design. On the other hand, software quality heavily depends on the ability to upgrade and patch source code to produce iterative releases. Even though blockchain-based software profits significantly from the technology’s immutability, still a certain degree of mutability is needed for bug fixing and potential product improvements.

In this post, we are going to learn:

Why do we need to upgrade smart contracts?
Understanding how upgrades work under the hood?
Using OpenZeppelin CLI to easily write/manage "upgradable" smart contracts.
Upgrading Contracts Programmatically using the OpenZeppelin Upgrades library.
A few Limitations & Workarounds of Upgradable Contracts

If you are just looking for a way to write upgradable contracts and don't want to go through "how this all works", then just head over to the 3rd section.

In case you get stuck somewhere or have any doubts, let us know here.

Why do we need to upgrade?

Smart contracts in Ethereum are immutable by default. Once you create them there is no way to alter them, effectively acting as an unbreakable contract among participants.

However, there are several scenarios where we wish if there was a way to upgrade the contracts. There are a lot of examples in which millions of dollars worth of Ether were stolen/hacked, which could be saved if we could update the smart contracts.

How do upgrades work under the hood?

There can be several ways we can upgrade our contracts.

The most obvious way will be something like this:

Create & deploy a new version of the contract.
Manually migrate all states from the old contract to the new contract.

This seems to work, but has several problems.

Migrating the contract state can be expensive.
As we create & deploy a new contract, the contract address will change. So you would need to update all contracts that interacted with the old contract to use the address of the new version.
You would also have to reach out to all your users and convince them to start using the new contract and handle both contracts being used simultaneously, as users are slow to migrate.

A better way is to use a proxy contract with an interface where each method delegates to the implementation contract (which contains all the logic).

A delegate call is similar to a regular call, except that all code is executed in the context of the caller (proxy), not of the callee (implementation). Because of this, a transfer in the implementation contract’s code will transfer the proxy’s balance, and any reads or writes to the contract storage will read or write from the proxy’s storage.

This approach is better because the users only interact with the proxy contract and we can change the implementation contract while keeping the same proxy contract.

This seems better than the previous approach, but if we need to make any changes to the implementation contract methods, we would need to update the proxy contract's methods too (as the proxy contract has interface methods). Hence, users will need to change the proxy address.

To solve this problem, we can use a fallback function in the proxy contract. The fallback function will execute on any request, redirecting the request to the implementation and returning the resulting value (using opcodes). This is similar to the previous approach, but here the proxy contract doesn’t have interface methods, only a fallback function, so there is no need to change the proxy address if contract methods are changed.

This was a basic explanation which is enough for us to work with upgradable contracts. In case, you want to dig deep into proxy contract code and different proxy patterns, then check out this post.

How Upgradable Smart Contracts Work Under the Hood

OpenZeppelin Upgrades

As we saw above, there are a lot of things you need to manage while writing upgradeable contracts.

Fortunately, projects like OpenZeppelin have built CLI tools & libraries, which provide an easy to use, simple, robust, and opt-in upgrade mechanism for smart contracts that can be controlled by any type of governance, be it a multi-sig wallet, a simple address or a complex DAO.

Let's first build a basic upgradeable contract using the OpenZeppelin CLI tool. You can find the code for the below implementation here.

In case you get stuck somewhere or have any doubts, let us know here.

OpenZeppelin Upgrades CLI

Working with OpenZeppelin CLI requires Node.js for development. If you don't have it already, install node using whatever package manager you prefer or using the official installer.

Project Setup

Create a folder named upgradable-smart-contracts and go inside the folder.

$ mkdir upgradable-smart-contracts && cd upgradable-smart-contracts

Throughout this tutorial, we use the *$* character to indicate your terminal’s shell prompt. When following along, don’t type the *$* character, or you’ll get some weird errors.

We will use a local blockchain for this tutorial. The most popular local blockchain is Ganache. To install and run it on your project, run:

$ npm install --save-dev ganache-cli && npx ganache-cli --deterministic

Now, start a new shell/terminal in the same folder run the following command to install the CLI tool:

$ npm install --save-dev @openzeppelin/cli

To manage your deployed contracts, you need to create a new CLI project. Run the following command, and provide it with a name and version number for your project when prompted:

$ npx openzeppelin init

Two things will happen during initialization. First, an .openzeppelin directory will be created, holding project-specific information. This directory will be managed by the CLI: you won’t need to manually edit anything. You should, however, commit some of these files to Git.

Second, the CLI will store network configuration in a file called networks.js. For convenience, it is already populated with an entry called development, with configuration matching Ganache's default.

You can see all the unlocked accounts by running the following command:

$ npx openzeppelin accounts

List of unlocked accounts

Writing & deploying contracts

Now, let's create a contract named TodoList in the contracts folder.

// contracts/TodoList.sol
pragma solidity ^0.6.3;

contract TodoList {
    string[] private list;

    // Emitted when the storeda new item is added to the list
    event ItemAdded(string item);

    // Adds a new item in the list
    function addItem(string memory newItem) public {
        list.push(newItem);
        emit ItemAdded(newItem);
    }

    // Gets the item from the list according to index
    function getListItem(uint256 index)
        public
        view
        returns (string memory item)
    {
        return list[index];
    }
}

Now, let's deploy this contract on the local blockchain.

$ npx openzeppelin create

As we can see our contract is deployed at 0xD833215cBcc3f914bD1C9ece3EE7BF8B14f841bb.

Let's add an item ("responding to emails") to the list array using the addItem() function by running npx openzeppelin send-tx.

Now, let's suppose we need to add a new function named getListSize() to get the size of the list. Just add a new function inside the TodoList contract.

// contracts/TodoList.sol
pragma solidity ^0.6.3;

contract TodoList {
    // ...


    // Gets the size of the list
    function getListSize() public view returns (uint256 size) {
        return list.length;
    }
}

After changing the Solidity file, we can now just upgrade the instance we had deployed earlier by running the openzeppelin upgrade command.

Done! Our TodoList instance has been upgraded to the latest version of the code *while keeping its state and the same address as before*. We didn’t need to create & deploy the proxy contract or link the proxy to the TodoList. All that is done under the hood!

Let’s try it out by invoking the new getListSize() function, and checking the size of the list in the new contract:

That’s it! Notice how the size of the list was preserved throughout the upgrade, as well as its address. And this process is the same regardless of whether you are working on a local blockchain, a testnet, or the main network.

Upgrading Contracts Programmatically

If you want to create and upgrade contracts from your JavaScript code instead of via the command line, you can use the *OpenZeppelin Upgrades* library instead of the CLI.

The CLI does not just manage contract upgrades, but also compilation, interaction, and source code verification. The Upgrades library only takes care of creating and upgrading. The library also does not keep track of the contracts you have already deployed, nor runs any initializer or storage layout validations, as the CLI does. Nevertheless, these capabilities may be added to the Upgrades library in the near future.

You can find the code for the below implementation here.

In case you get stuck somewhere or have any doubts, let us know here.

In case you have not followed the above OpenZeppelin CLI part, you need to install NodeJs & Ganache as instructed here.

Your first step will be to install the library in your project, and you will also probably want to install web3 to interact with our contracts using JavaScript, and @openzeppelin/contract-loader to load the contracts from the JSON artifacts.

$ npm install web3 @openzeppelin/upgrades @openzeppelin/contract-loader

Now, create a file index.js inside upgradable-smart-contracts folder and paste this boilerplate code.

// index.js
const Web3 = require("web3");
const {
  ZWeb3,
  Contracts,
  ProxyAdminProject
} = require("@openzeppelin/upgrades");

async function main() {
  // Set up web3 object, connected to the local development network, initialize the Upgrades library
  const web3 = new Web3("http://localhost:8545");
  ZWeb3.initialize(web3.currentProvider);
  const loader = setupLoader({ provider: web3 }).web3;
}

main();

Here we set up web3 object, connected to the local development network, initialize the Upgrades library via ZWeb3.initialize, and initialize the contract loader.

Now, add this following snippet in the main() to create a new project, to manage our upgradeable contracts.

async function main() {
  // ...

  //Fetch the default account
  const from = await ZWeb3.defaultAccount();

  //creating a new project, to manage our upgradeable contracts.
  const project = new ProxyAdminProject("MyProject", null, null, {
    from,
    gas: 1e6,
    gasPrice: 1e9
  });
}

Now, using this project, we can create an instance of any contract. The project will take care of deploying it in such a way it can be upgraded later.

Let's create 2 contracts, TodoList1 and its updated version TodoList2 inside upgradable-smart-contracts/contracts folder.

// contracts/TodoList1.sol
pragma solidity ^0.6.3;

contract TodoList1 {
    string[] private list;

    // Emitted when the storeda new item is added to the list
    event ItemAdded(string item);

    // Adds a new item in the list
    function addItem(string memory newItem) public {
        list.push(newItem);
        emit ItemAdded(newItem);
    }

    // Gets the item from the list according to index
    function getListItem(uint256 index)
        public
        view
        returns (string memory item)
    {
        return list[index];
    }
}

To create TodoList2, just add a new getListSize() function in the above contract.

// contracts/TodoList2.sol
pragma solidity ^0.6.3;

contract TodoList2 {
    string[] private list;

    // Emitted when the storeda new item is added to the list
    event ItemAdded(string item);

    // Adds a new item in the list
    function addItem(string memory newItem) public {
        list.push(newItem);
        emit ItemAdded(newItem);
    }

    // Gets the item from the list according to index
    function getListItem(uint256 index)
        public
        view
        returns (string memory item)
    {
        return list[index];
    }

    // Gets the size of the list
    function getListSize() public view returns (uint256 size) {
        return list.length;
    }
}

Now, we need to compile these 2 contracts using:

$ npx openzeppelin compile

This will create JSON contract artifacts in the build/contracts folder. These artifacts files contain all the information about the contracts that we would need to deploy and interact with the contracts.

Now, let's create an instance of TodoList1 using the project we created above.

async function main() {
//...


//Using this project, we can now create an instance of any contract.
  //The project will take care of deploying it in such a way it can be upgraded later.
  const TodoList1 = Contracts.getFromLocal("TodoList1");
  const instance = await project.createProxy(TodoList1);
  const address = instance.options.address;
  console.log("Proxy Contract Address 1: ", address);
}

Here we get the TodoList1 contract details from the contract artifacts we created above using Contracts.getFromLocal. Then we create & deploy a pair of proxy & implementation (TodoList1) contracts and link the proxy contract to the TodoList1 via project.createProxy method. Finally, we print out the address of our proxy contract.

Now, let's add an item to the list using addItem() method and then fetch the added item using getListItem().

async function main() {
//...

  // Send a transaction to add a new item in the TodoList1
  await todoList1.methods
    .addItem("go to class")
    .send({ from: from, gas: 100000, gasPrice: 1e6 });

  // Call the getListItem() function to fetch the added item from TodoList1
  var item = await todoList1.methods.getListItem(0).call();
  console.log("TodoList1: List Item 0: ", item);
}

Now, let's update our TodoList1 contract to TodoList2.

async function main() {
//...


//After deploying the contract, you can upgrade it to a new version of
  //the code using the upgradeProxy method, and providing the instance address.
  const TodoList2 = Contracts.getFromLocal("TodoList2");
  const updatedInstance = await project.upgradeProxy(address, TodoList2);
  console.log("Proxy Contract Address 2: ", updatedInstance.options.address);
}

Here we get the TodoList2 contract details from the contract artifacts. Then we update our contract via project.upgradeProxy method which takes 2 parameters, the address of the proxy contract that we deployed in the previous step, and the TodoList2 contract object. We then print out the address of the proxy contract after the update.

Now, let's add a new item to the TodoList2 and fetch the items.

async function main() {
//...


  // Send a transaction to add a new item in the TodoList2
  await todoList2.methods
    .addItem("code")
    .send({ from: from, gas: 100000, gasPrice: 1e6 });

  // Call the getListItem() function to fetch the added items from TodoList2
  var item0 = await todoList2.methods.getListItem(0).call();
  var item1 = await todoList2.methods.getListItem(1).call();
  console.log("TodoList2: List Item 0: ", item0);
  console.log("TodoList2: List Item 1: ", item1);
}

Now, let's run the index.js using node index.js.

Here we can observe 2 things:

The address of the proxy contract did not change even after we updated TodoList1 to TodoList2.
As we got 2 items from the TodoList2, this shows that the state was preserved throughout the update.

Hence we can say that TodoList1 instance has been upgraded to the latest version of the code (TodoList2), *while keeping its state and the same address as before*.

In case you get stuck somewhere or have any doubts, let us know here.

Now, as we have seen how to upgrade contracts, let's see a few limitations & workarounds that you need to know about when writing more complex contracts.

A few things to keep in mind: limitations & workarounds

When working with upgradeable contracts using OpenZeppelin Upgrades, there are a few minor caveats to keep in mind when writing your Solidity code.

It’s worth mentioning that these restrictions have their roots in how the Ethereum VM works, and apply to all projects that work with upgradeable contracts, not just OpenZeppelin Upgrades.

If your contract is not compatible for upgrades, the CLI will warn you when you try to upgrade. This feature is not there yet in the OpenZeppelin Upgrades libraries.

In order to understand the limitations & workarounds, let's take an Example contract, explore the limitations in the contract and add some workarounds to make the contract upgradable.

// contracts/Example.sol

pragma solidity ^0.6.0;

import "github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC20/ERC20Capped.sol";

contract Example {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    constructor(uint8 cap) public {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Limitation 1: No Constructors

Due to a requirement of the proxy-based upgradeability system, no constructors can be used in upgradeable contracts. To learn about the reasons behind this restriction, head to this post.

Workaround: Initializer

A workaround is to replace the constructor with a function, typically named initialize, where you run constructor logic.

// contracts/Example.sol
pragma solidity ^0.6.0;

import "github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC20/ERC20Capped.sol";

contract Example {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    function initialize(uint8 cap) public {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Now, as the constructor is called only once when the contract is initialized, we need to add a check to ensure that the initialize function gets called only once.

// contracts/Example.sol
pragma solidity ^0.6.0;

import "github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC20/ERC20Capped.sol";

contract Example {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    bool private _initialized = false;

    function initialize(uint8 cap) public {
        require(!_initialized);
        _initialized = true;
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Since this will be a common thing to do when writing upgradeable contracts, OpenZeppelin Upgrades provides an Initializable base contract that has an initializer modifier that takes care of this:

// contracts/Example.sol
pragma solidity ^0.6.0;

import "github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC20/ERC20Capped.sol";
import "@openzeppelin/upgrades/contracts/Initializable.sol";

contract Example is Initializable {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    function initialize(uint8 cap) public initializer {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Another difference between a constructor and a regular function is that Solidity takes care of automatically invoking the constructors of all ancestors of a contract. When writing an initializer, you need to take special care to manually call the initializers of all parent contracts:

// contracts/Example.sol
pragma solidity ^0.6.0;

import "github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC20/ERC20Capped.sol";
import "@openzeppelin/upgrades/contracts/Initializable.sol";


contract BaseExample is Initializable {
    uint256 public createdAt;

    function initialize() initializer public {
        createdAt = block.timestamp;
    }

}

contract Example is BaseExample {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    function initialize(uint8 cap) initializer public {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Keep in mind that this restriction affects not only your contracts, but also the contracts you import from a library. Consider for example ERC20Capped from OpenZeppelin Contracts: the contract initializes the token’s cap in its constructor.

pragma solidity ^0.6.0;

import "./ERC20.sol";

/**
 * @dev Extension of {ERC20} that adds a cap to the supply of tokens.
 */
contract ERC20Capped is ERC20 {
    uint256 private _cap;

    /**
     * @dev Sets the value of the `cap`. This value is immutable, it can only be
     * set once during construction.
     */
    constructor (uint256 cap) public {
        require(cap > 0, "ERC20Capped: cap is 0");
        _cap = cap;
    }

    //...
}

This means you should not be using these contracts in your OpenZeppelin Upgrades project. Instead, make sure to use @openzeppelin/contracts-ethereum-package, which is an official fork of OpenZeppelin Contracts that has been modified to use initializers instead of constructors. Take a look at how ERC20Capped looks in @openzeppelin/contracts-ethereum-package:

pragma solidity ^0.5.0;

import "@openzeppelin/upgrades/contracts/Initializable.sol";
import "./ERC20Mintable.sol";

/**
 * @dev Extension of {ERC20Mintable} that adds a cap to the supply of tokens.
 */
contract ERC20Capped is Initializable, ERC20Mintable {
    uint256 private _cap;

    /**
     * @dev Sets the value of the `cap`. This value is immutable, it can only be
     * set once during construction.
     */
    function initialize(uint256 cap, address sender) public initializer {
        ERC20Mintable.initialize(sender);

        require(cap > 0, "ERC20Capped: cap is 0");
        _cap = cap;
    }

    //...
}

Whether using OpenZeppelin Contracts or another Ethereum Package, always make sure that the package is set up to handle upgradeable contracts.

// contracts/Example.sol
pragma solidity ^0.6.0;

import "@openzeppelin/contracts-ethereum-package/contracts/token/ERC20/ERC20Capped.sol";
import "@openzeppelin/upgrades/contracts/Initializable.sol";


contract BaseExample is Initializable {
    uint256 public createdAt;

    function initialize() initializer public {
        createdAt = block.timestamp;
    }

}

contract Example is BaseExample {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    function initialize(uint8 cap) initializer public {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Limitation 2: Initial Values in Field Declarations

Solidity allows defining initial values for fields when declaring them in a contract.

contract Example is BaseExample {
    uint256 private _cap = 1000000000000000000;

    //...
}

This is equivalent to setting these values in the constructor, and as such, will not work for upgradeable contracts.

Workaround: Initializer

Make sure that all initial values are set in an initializer function as shown below; otherwise, any upgradeable instances will not have these fields set.

//...

contract Example is BaseExample {
    uint256 private _cap;
    ERC20Capped public token;

    function initialize(uint8 cap) initializer public {
        _cap = 1000000000000000000;
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Note that it still is fine to set constants here, because the compiler does not reserve a storage slot for these variables, and every occurrence is replaced by the respective constant expression. So the following still works with OpenZeppelin Upgrades:

//...

contract Example is BaseExample {
    uint256 constant private _cap = 1000000000000000000;

    //...
}

Limitation: Creating New Instances From Your Contract Code

When creating a new instance of a contract from your contract’s code, these creations are handled directly by Solidity and not by OpenZeppelin Upgrades, which means that *these contracts will not be upgradeable*.

For instance, in the following example, even if Example is upgradeable (if created via openzeppelin create Example), the token contract created is not:

//...

contract Example is BaseExample {
    uint256 private _cap = 1000000000000000000;
    ERC20Capped public token;

    function initialize(uint8 cap) initializer public {
        _cap = cap;
        token = new ERC20Capped(_cap);
    }
}

Workaround: Inject a pre-deployed contract from CLI

The easiest way around this issue is to avoid creating contracts on your own altogether: instead of creating a contract in an initialize function, simply accept an instance of that contract as a parameter, and inject it after creating it from the OpenZeppelin CLI:

//...

contract Example is BaseExample {
    ERC20Capped public token;

    function initialize(ERC20Capped _token) initializer public {
        token = _token;
    }
}

$ TOKEN=$(npx openzeppelin create TokenContract)
$ npx oz create Example --init --args $TOKEN

Workaround: OpenZeppelin App Contract

An advanced alternative, if you need to create upgradeable contracts on the fly, is to keep an instance of your OpenZeppelin project’s App in your contracts. The App is a contract that acts as the entrypoint for your OpenZeppelin project, which has references to your logic implementations, and can create new contract instances:

// contracts/Example.sol
pragma solidity ^0.6.0;

import "@openzeppelin/upgrades/contracts/Initializable.sol";
import "@openzeppelin/upgrades/contracts/application/App.sol";

contract BaseExample is Initializable {
    //...
}

contract Example is BaseExample {

  App private app;

  function initialize(App _app) initializer public {
    app = _app;
  }

  function createNewToken() public returns(address) {
    return app.create("@openzeppelin/contracts-ethereum-package", "ERC20Capped");
  }
}

Potentially Unsafe Operations

When working with upgradeable smart contracts, you will always interact with the proxy contract instance, and never with the underlying logic (implementation) contract. However, nothing prevents a malicious actor from sending transactions to the logic contract directly. This does not pose a threat, since any changes to the state of the logic contracts do not affect your proxy contract instances, as the storage of the logic contracts is never used in your project.

There is, however, an exception. If the direct call to the logic contract triggers a selfdestruct operation, then the logic contract will be destroyed, and all your contract instances will end up delegating all calls to an address without any code. This would effectively break all contract instances in your project.

A similar effect can be achieved if the logic contract contains a delegatecall operation. If the contract can be made to delegatecall into a malicious contract that contains a selfdestruct, then the calling contract will be destroyed.

pragma solidity ^0.6.0;

// The Exmaple contract makes a `delegatecall` to the Malicious contract. Thus, even if the Malicious contract runs the `selfdestruct` function, it is run in the context of the Example contract, thus killing the Example contract.  

contract Example {
    function testFunc(address malicious) public {
        malicious.delegatecall(abi.encodeWithSignature("kill()"));
    }
}

contract Malicious {
    function kill() public {
        address payable addr = address(uint160(address(0x4Bf8c809c898ee52Eb7fc6e1FdbB067423326B2A)));
        selfdestruct(addr);
    }
}

As such, it is strongly recommended to avoid any usage of either selfdestruct or delegatecall in your contracts. If you need to include them, make absolutely sure they cannot be called by an attacker on an uninitialized logic contract.

Modifying Your Contracts

When writing new versions of your contracts, either due to new features or bugfixing, there is an additional restriction to observe: you cannot change the order in which the contract state variables are declared, nor their type. You can read more about the reasons behind this restriction by learning about Proxies.

Violating any of these storage layout restrictions will cause the upgraded version of the contract to have its storage values mixed up, and can lead to critical errors in your application.

This means that if you have an initial contract that looks like this:

pragma solidity ^0.6.3;

contract Example {
    string public tokenName;
    uint8 public decimals;
}

Then you cannot change the type of a variable:

pragma solidity ^0.6.3;

contract Example {
    string public tokenName;
    uint public decimals;
}

Or change the order in which they are declared:

pragma solidity ^0.6.3;

contract Example {
    uint public decimals;
    string public tokenName;
}

Or introduce a new variable before existing ones:

pragma solidity ^0.6.3;

contract Example {
    string public tokenSymbol;
    string public tokenName;
    uint public decimals;
}

Or remove an existing variable:

pragma solidity ^0.6.3;

contract Example {
    string public tokenName;
}

If you need to introduce a new variable, make sure you always do so at the end:

pragma solidity ^0.6.3;

contract Example {
    string public tokenName;
    uint public decimals;
    string public tokenSymbol;
}

Keep in mind that if you rename a variable, then it will keep the same value as before after upgrading. This may be the desired behaviour if the new variable is semantically the same as the old one:

pragma solidity ^0.6.3;

contract Example {
    string public tokenName;
    uint public decimalCount;   // starts with the value of `decimals`
}

And if you remove a variable from the end of the contract, note that the storage will not be cleared. A subsequent update that adds a new variable will cause that variable to read the leftover value from the deleted one.

pragma solidity ^0.6.3;

contract Example1 {
    string public tokenName;
    uint public decimals;
}

// Updating Example1 --> Example2

contract Example2 {
    string public tokenName;
}

// Updating Example2 --> Example3

contract Example3 {
    string public tokenName;
    uint public decimalCount;   // starts with the value of `decimals`
}

Note that you may also be inadvertently changing the storage variables of your contract by changing its parent (base) contracts. For instance, if you have the following contracts:

pragma solidity ^0.6.3;

contract BaseExample1 {
    uint256 createdAt;
}

contract BaseExample2 {
    string version;
}

contract Example is BaseExample1, BaseExample2 {}

Then modifying Example by swapping the order in which the base contracts are declared, or adding new base contracts or removing base contracts, will change how the variables are actually stored:

pragma solidity ^0.6.3;

contract BaseExample1 {
    uint256 createdAt;
}

contract BaseExample2 {
    string version;
}

//swapping the order in which the base contracts are declared
contract Example is BaseExample2, BaseExample1 {}

//Or...

//removing base contract(s)
contract Example is BaseExample1 {}

//Or...

contract BaseExample3 {} 

//adding new base contract
contract Example is BaseExample1, BaseExample2, BaseExample3 {}

You also cannot add new variables to base contracts, if the child has any variables of its own. Given the following scenario:

pragma solidity ^0.6.3;

contract BaseExample {}

contract Example is BaseExample {
    string tokenName;
}

//Now, if the BaseExample is updated to the following

contract BaseExample {
    string version;     // takes the value of `tokenName` 
}

contract Example is BaseExample {
    string tokenName;
}

Then the variable version would be assigned the slot that tokenName had in the previous version.

You also remove a variable from the base contract, if the child has any variables of its own. For example:

pragma solidity ^0.6.3;

contract BaseExample {
    uint256 createdAt;
    string version;
}

contract Example is BaseExample {
    string tokenName;
}

//Now, if the BaseExample is updated to the following

contract BaseExample {
    uint256 createdAt; 
}

contract Example is BaseExample {
    string tokenName;   //takes the value of `version`
}

Here, as we remove the version variable from the BaseExample, the memory slot for version (before update) will now be used by tokenName (after update).

A workaround for this is to declare unused variables on base contracts that you may want to extend in the future, as a means of "reserving" those slots. So, basically, keeping the number and order of the variables in the parent and child contracts same for all the updates.

pragma solidity ^0.6.3;

contract BaseExample {
    string someVar1;
    string someVar2;
    string someVar3;

    //...
}

Note that this trick does not involve increased gas usage.

In case you get stuck somewhere or have any doubts, let us know here.

References

Tutorial: Building a Twitter Bot using IPFS Cluster 🤖

vasa — Sat, 25 Jan 2020 23:35:45 +0000

Till now, we have explored a number of fun examples using InterPlanetary File System (IPFS), InterPlanetary Linked Data (IPLD) & Libp2p. We have built Websites on IPFS, Youtube on IPFS, Online Publication on IPLD & Chat Application on Libp2p.

In this tutorial we are going to build a Twitter Bot(or a Pinbot) using IPFS
Cluster.

You can see our Twitter bot in action!

You can try it out yourself.

Follow the @simpleaswater_ twitter account so that bot can filter your tweets out.
Tweet a photo, gif, video or any media, and mention “@simpleaswater_” in your tweet. Just like below:

The bot will reply with the links to your content stored on the IPFS Cluster Network.

Cool, right? Let’s see how you can build your own twitter pinbot!

In this tutorial, we are go through:

An Intro to IPFS Cluster, its significance & use-cases.
Setting up an IPFS Cluster Network.
Setting up Twitter Developer Account.
Building a Twitter bot.

You can find the full code implementation here.

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

Intro to IPFS Cluster

Before setting up our own IPFS Cluster network, it’s good to spend a few minutes to understand why do we need IPFS Cluster and what is it.

If you are new to InterPlanetary File System (IPFS) then we would recommend you to check out this post first.

If you are familiar with IPFS, then you already know that IPFS aims to make the internet decentralized using content addressing approach.

The way this works is similar to services like BitTorrent where every user does not just consume the data but also serves it to other people in the BitTorrent network. That means you can not only get the data(let’s say Star Wars movie) from anyone on the network who has it, but also share this data with anyone who asks for it on the network.

But if all the people who have the Star Wars movie go offline, then you are out of luck.

No movie for you 😔

Now to understand why do we need IPFS Cluster consider the following scenario:

Let’s suppose you want to store your collection of favorite songs on the IPFS Network. You upload your playlist on the IPFS network and the people who also love the same songs can get the songs from you.Now, you may also have other things on your device that you want to keep(your Movies, vacation photos, etc.). So, you decide to remove the playlist from your device. After all, you can get your playlist from the people who downloaded it from you earlier. So, you are not worried about your songs getting lost, and go to sleep happily.One fine day, you want to get your songs back for your long workout session but are devastated to see that you can’t find your playlist on the IPFS Network 😨But how can this happen? You saved it on the network, right?It turns out that all the people who saved the playlist are either offline or have deleted the songs, as they got bored with them.The problem was that our playlist was not stored redundantly on a number of nodes(or devices) so that we could be sure that it would not get lost in the oblivion.

So, how do we ensure data availability and redundancy on IPFS?

Enter IPFS Cluster.

The purpose of IPFS Cluster is to make it easy for you to orchestrate/manage data across several IPFS peers by allocating, replicating and tracking a global pinset(list of saved data) distributed among multiple peers.

This way you can build a network of IPFS peers which redundantly store data for us.

If you want to know more about IPFS Cluster and see how it works under the hood, then head here.

If you have any queries/doubts till now, then feel free to reach us out on our discord channel.

Now, as we have explored why we need IPFS Cluster let’s setup our own IPFS Cluster network, on which we will save our tweets.

Setting up Your IPFS Peer

As IPFS Cluster network is a collection of IPFS peers, first we need to install & run an IPFS peer.

You can follow this guide to install IPFS Peer on your device.

After installing the IPFS, you can test your installation using ipfs version

$ ipfs version
ipfs version <VERSION_NUMBER>

$ ipfs help
USAGE:

ipfs - Global p2p merkle-dag filesystem.
...

NOTE: Throughout this tutorial, we use the *$** character to indicate your terminal’s shell prompt. When following along, don’t type the $ character,or you’ll get some weird errors.*

Now, let’s initialize your IPFS peer using ipfs init

$ ipfs init

initializing IPFS node at /home/vasa/.ipfs
generating 2048-bit RSA keypair...done
peer identity: Qmcpo2iLBikrdf1d6QU6vXuNb6P7hwrbNPW9kLAH8eG67z
to get started, enter:

ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme

Voila! Your IPFS peer setup is complete!

If you are having any problems while setting up your IPFS peer, then feel free to reach us out on our discord channel.

Setting up Your IPFS Cluster Peer

In order to run an IPFS Cluster peer we need to install ipfs-cluster-service.

First, download theipfs-cluster-service binary for your Operating System.
Now, unzip/untar the downloaded binary and you will find a ipfs-cluster-service executable inside the folder. Place this binary where it can be run by a system user(usually usr/local/bin).
Now run the following command to initialize generate the IPFS Cluster configuration file.

$ ipfs-cluster-service init

Viola! You have setup your IPFS Cluster Peer.

Making a few changes to IPFS Cluster Config

In order to make the twitter bot example work, we need to change the IPFS Cluster config a bit. If you want to know why we are doing these changes, then you can refer to this discussion thread.

To make the changes, first open your IPFS Cluster config located at $HOME/.ipfs-cluster/service.json.

Now, scroll down to the api JSON object. In the api section, you will find restapi JSON object. You need to add these 3 key-values: libp2p_listen_multiaddress, private_key and id as shown below.

{
...
"api": {
    "ipfsproxy": {
        ...
    },
    "restapi": {
      "libp2p_listen_multiaddress": "/ip4/127.0.0.1/tcp/9696",
      "private_key": "CAASqAkwggSkAgEAAoIBAQDLZZcGcbe4urMBVlcHgN0fpBymY+xcr14ewvamG70QZODJ1h9sljlExZ7byLiqRB3SjGbfpZ1FweznwNxWtWpjHkQjTVXeoM4EEgDSNO/Cg7KNlU0EJvgPJXeEPycAZX9qASbVJ6EECQ40VR/7+SuSqsdL1hrmG1phpIju+D64gLyWpw9WEALfzMpH5I/KvdYDW3N4g6zOD2mZNp5y1gHeXINHWzMF596O72/6cxwyiXV1eJ000k1NVnUyrPjXtqWdVLRk5IU1LFpoQoXZU5X1hKj1a2qt/lZfH5eOrF/ramHcwhrYYw1txf8JHXWO/bbNnyemTHAvutZpTNrsWATfAgMBAAECggEAQj0obPnVyjxLFZFnsFLgMHDCv9Fk5V5bOYtmxfvcm50us6ye+T8HEYWGUa9RrGmYiLweuJD34gLgwyzE1RwptHPj3tdNsr4NubefOtXwixlWqdNIjKSgPlaGULQ8YF2tm/kaC2rnfifwz0w1qVqhPReO5fypL+0ShyANVD3WN0Fo2ugzrniCXHUpR2sHXSg6K+2+qWdveyjNWog34b7CgpV73Ln96BWae6ElU8PR5AWdMnRaA9ucA+/HWWJIWB3Fb4+6uwlxhu2L50Ckq1gwYZCtGw63q5L4CglmXMfIKnQAuEzazq9T4YxEkp+XDnVZAOgnQGUBYpetlgMmkkh9qQKBgQDvsEs0ThzFLgnhtC2Jy//ZOrOvIAKAZZf/mS08AqWH3L0/Rjm8ZYbLsRcoWU78sl8UFFwAQhMRDBP9G+RPojWVahBL/B7emdKKnFR1NfwKjFdDVaoX5uNvZEKSl9UubbC4WZJ65u/cd5jEnj+w3ir9G8n+P1gp/0yBz02nZXFgSwKBgQDZPQr4HBxZL7Kx7D49ormIlB7CCn2i7mT11Cppn5ifUTrp7DbFJ2t9e8UNk6tgvbENgCKXvXWsmflSo9gmMxeEOD40AgAkO8Pn2R4OYhrwd89dECiKM34HrVNBzGoB5+YsAno6zGvOzLKbNwMG++2iuNXqXTk4uV9GcI8OnU5ZPQKBgCZUGrKSiyc85XeiSGXwqUkjifhHNh8yH8xPwlwGUFIZimnD4RevZI7OEtXw8iCWpX2gg9XGuyXOuKORAkF5vvfVriV4e7c9Ad4Igbj8mQFWz92EpV6NHXGCpuKqRPzXrZrNOA9PPqwSs+s9IxI1dMpk1zhBCOguWx2m+NP79NVhAoGBAI6WSoTfrpu7ewbdkVzTWgQTdLzYNe6jmxDf2ZbKclrf7lNr/+cYIK2Ud5qZunsdBwFdgVcnu/02czeS42TvVBgs8mcgiQc/Uy7yi4/VROlhOnJTEMjlU2umkGc3zLzDgYiRd7jwRDLQmMrYKNyEr02HFKFn3w8kXSzW5I8rISnhAoGBANhchHVtJd3VMYvxNcQb909FiwTnT9kl9pkjhwivx+f8/K8pDfYCjYSBYCfPTM5Pskv5dXzOdnNuCj6Y2H/9m2SsObukBwF0z5Qijgu1DsxvADVIKZ4rzrGb4uSEmM6200qjJ/9U98fVM7rvOraakrhcf9gRwuspguJQnSO9cLj6",
      "id": "QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm",
      ...
      }
  }
  ...
}

NOTE: The *private_key that we are using is just for the demo purpose. If you are building a production bot, you should create your own libp2p private_key and id.*

Starting Your IPFS Cluster Network

Now, as we have installed IPFS and IPFS Cluster peers, we can start our Cluster network.

Open a terminal window and start the IPFS peer using ipfs daemon
Open **another terminal **and start the IPFS Cluster peer using ipfs-cluster-service daemon

Running IPFS daemon

Open **another terminal **and start the IPFS Cluster peer using ipfs-cluster-service daemon

Running IPFS Cluster daemon

If you managed to reach here, then you deserve a pat on your back!

You are now running a IPFS Cluster network with 1 peer. For the purpose of this tutorial we will proceed with this setup. In case, you want to setup a more complex production infrastructure, then you can refer to IPFS Cluster docs.

If you have any queries/doubts, then feel free to reach us out on our **discord channel.**

Creating a Twitter Developer Account

To build a twitter bot, first we need to setup a developer account on twitter.

Go to Twitter developers website. Click the “Create an app” button.

You will be prompted to apply for the twitter developer account.

Now, select the “Making a bot” option for your primary reason for using twitter developer tools, and click “Next”.

In the next page, you will be asked some personal details. Fill them up and click “Next”.
In the next page you will be asked a few more questions about how will you use twitter’s APIs. Fill up the answers by saying that you are creating a bot for fun 😉. After filling all the info, click “Next”.
In the last page, you will be asked to confirm the details that you provided.
You may have to confirm your email account. And wait for sometime to get your account approved.

After your account has been approved, you can go your twitter apps page, and click “Create an app”, fill the app details and hit “Create” button.

Congratulations! You now have a twitter app 🎉

Now, come back to the twitter apps page, where you can see your app. Click on the “details” button.

Navigate to the “keys and tokens” section, where you will find all the credentials we want. Copy the “API key”, “API secret key”, “Access token” & “Access token secret”.

Building the Twitter Bot 🤖

Now, as we have everything we need to build the Twitter Pinbot, let’s fire up our code editor!

Installing Golang

We are going to use Golang to build our Twitter bot. You can download Golang from here.

First you need to download Golang.
You will be redirected to a page with installation instructions. Follow these instructions to install Golang.

After installing Golang, we are ready to start with the twitter bot code.

Writing the Twitter Bot Code 👩‍💻👨‍💻

We are going to build a twitter bot which works in the following way:

You need to follow the twitter bot account, so that the bot can filter out your tweets from all the tweets.
Then you can “pin”, “unpin” or “add” data to the IPFS Cluster Network by tweeting to @botHandle with some specific text.

//To pin a CID with a name/label to IPFS Cluster Network
@botHandle !pin <cid> <name>

//To unpin a CID from IPFS Cluster Network
@botHandle !unpin <cid>

//To add a file(via URL) to IPFS Cluster Network
@botHandle !add <url-to-single-file>

//To check out what the bot can do
@botHandle !help

//Adding photos, memes, videos to IPFS Cluster Network
Tweet to @botHandle attaching photos, memes or videos

Now, let’s start by downloading some boilerplate code.

Clone the *boilerplate *branch of twitter-pinbot repo.

$ git clone --single-branch --branch boilerplate https://github.com/simpleaswater/twitter-pinbot

Now, first we need to make a few changes in the *config.json *file.

{
  "twitter_name": "@botHandle",
  "twitter_id": "@botHandle",
  "consumer_key": "API key",
  "consumer_secret": "API secret key",
  "access_key": "Access token",
  "access_secret": "Access token secret",
  "cluster_peer_addr": "cluster peer multiaddress",
  "cluster_username": "",
  "cluster_password": ""
}

You need to add your @botHandle and the credentials that we got from the twitter developer portal.

Then replace the “cluster peer multiaddress” with:

/ip4/127.0.0.1/tcp/9696/ipfs/QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm

If you look closely, you can see that this is the same multiaddress which we added above while changing the IPFS Cluster config file.

We do this to interact with the IPFS Cluster Setup that we created above.

You can leave the “cluster_username” & “cluster_password” blank.

Now, as we are done with our config.json *file, let’s start working on *main.go.

package main

import (
    "regexp"
)

// ConfigFile is the path of the default configuration file
var ConfigFile = "config.json"

// Gateway
var IPFSGateway = "https://ipfs.io"

const twittercom = "twitter.com"

type Action string

// Variables containing the different available actions
var (
    // (spaces)(action)whitespaces(arguments)
    actionRegexp = regexp.MustCompile(`^\s*([[:graph:]]+)\s+(.+)`)
    // (cid)whitespaces(name with whitespaces). [:graph:] does not
    // match line breaks or spaces.
    pinRegexp          = regexp.MustCompile(`([[:graph:]]+)\s+([[:graph:]\s]+)`)
    PinAction   Action = "!pin"
    UnpinAction Action = "!unpin"
    AddAction   Action = "!add"
    HelpAction  Action = "!help"
)

func main() {
    //Let's code ☕
}

The main.go file contains a few constants that we will be using as we move forward in the tutorial.

Let’s first add a way to read and manage the data from config.json.

// Config is the configuration format for the Twitter Pinbot
type Config struct {
    TwitterID       string `json:"twitter_id"`
    TwitterName     string `json:"twitter_name"`
    AccessKey       string `json:"access_key"`
    AccessSecret    string `json:"access_secret"`
    ConsumerKey     string `json:"consumer_key"`
    ConsumerSecret  string `json:"consumer_secret"`
    ClusterPeerAddr string `json:"cluster_peer_addr"`
    ClusterUsername string `json:"cluster_username"`
    ClusterPassword string `json:"cluster_password"`
}

// Function to read JSON config file
func readConfig(path string) *Config {
    cfg := &Config{}
    cfgFile, err := ioutil.ReadFile(path)
    if err != nil {
        log.Fatal(err)
    }
    err = json.Unmarshal(cfgFile, &cfg)
    if err != nil {
        log.Fatal(err)
    }
    return cfg
}

Here we have Config struct which manages different key-value pairs from our config.json file. Also, we have the function readConfig(path string) which takes the path of the config.json and parses the JSON from the file.

Next, we create the Bot struct, which is used to manage the different
functionalities of the twitter pinbot.

// Bot is a twitter bot which reads a user's timeline
// and performs actions on IPFS Cluster if the tweets
// match, i.e. a tweet with: "@botHandle !pin <cid> <name>"
// will pin something. The users with pin permissions are
// those who follow the bot. Retweets by users who follow
// the bot should also work. The bot will answer
// the tweet with a result.
type Bot struct {
    ctx    context.Context
    cancel context.CancelFunc

    name          string
    id            string
    twClient      *twitter.Client
    clusterClient client.Client

    followedBy sync.Map

    die chan struct{}
}

You will also have to import these 2 modules:

import (
    "github.com/dghubble/go-twitter/twitter"
    "github.com/ipfs/ipfs-cluster/api/rest/client"
)

Now, let’s add the function to create a Bot.

import (
    "github.com/multiformats/go-multiaddr"
    "github.com/dghubble/oauth1"
)

// New creates a new Bot with the Config.
func New(cfg *Config) (*Bot, error) {
    ctx, cancel := context.WithCancel(context.Background())

    // Creating Twitter client
    ocfg := oauth1.NewConfig(cfg.ConsumerKey, cfg.ConsumerSecret)
    token := oauth1.NewToken(cfg.AccessKey, cfg.AccessSecret)
    httpClient := ocfg.Client(ctx, token)
    twClient := twitter.NewClient(httpClient)

    // Creating IPFS Cluster client
    peerAddr, err := multiaddr.NewMultiaddr(cfg.ClusterPeerAddr)
    if err != nil {
        cancel()
        return nil, err
    }
    clusterClient, err := client.NewDefaultClient(&client.Config{
        APIAddr:  peerAddr,
        Username: cfg.ClusterUsername,
        Password: cfg.ClusterPassword,
        LogLevel: "info",
    })

    if err != nil {
        cancel()
        return nil, err
    }

    //Creating Bot Object
    bot := &Bot{
        ctx:           ctx,
        cancel:        cancel,
        twClient:      twClient,
        clusterClient: clusterClient,
        name:          cfg.TwitterName,
        id:            cfg.TwitterID,
        die:           make(chan struct{}, 1),
    }

    bot.fetchFollowers()
    go bot.watchFollowers()
    go bot.watchTweets()
    return bot, nil
}

Here we first create twClient which interacts with the twitter APIs. Also, we create clusterClient which interacts with our IPFS Cluster Setup. Then, using these 2, we create our Bot object.

At last, we have 3 methods:

fetchFollowers(): This method is used to fetch the list of bot followers, so that we can track their tweets, and respond accordingly.
watchFollowers(): This goroutine is used to watch the list of bot followers every 60 seconds. In case, somebody follows the bot, the list will be updated within 60 minutes, after which we will start filtering the tweets of the recent follower.
watchTweets(): This goroutine is used to watch & filter the tweets who mention our @botHandle.

We will implement these 3 methods in a while.

Next, we have a few small utility functions.

// Kill destroys this bot.
func (b *Bot) Kill() {
    b.cancel()
}

// Name returns the twitter handle used by the bot
func (b *Bot) Name() string {
    return b.name
}

// ID returns the twitter user ID used by the bot
func (b *Bot) ID() string {
    return b.id
}

As explained in the comments:

Kill(): destroys the bot.
Name(): returns the twitter handle used by the bot.
ID(): returns the twitter user ID used by the bot.

Now, let’s implement the above 3 methods.

//Fetching the bot Followers
func (b *Bot) fetchFollowers() {
    var nextCursor int64 = -1
    includeEntities := false
    for nextCursor != 0 {
        //Getting the List of bot followers
        followers, _, err := b.twClient.Followers.List(
            &twitter.FollowerListParams{
                Count:               200, //Number of bot followers we want in our list
                IncludeUserEntities: &includeEntities,
            })

        if err != nil {
            log.Println(err)
        }

        //Parsing and Storing the ID(s) of the bot followers
        for _, u := range followers.Users {
            _, old := b.followedBy.LoadOrStore(u.ID, struct{}{})
            if !old {
                //Printing the bot followers ScreenName
                log.Println("Friend: ", u.ScreenName)
            }
        }
        nextCursor = followers.NextCursor
        time.Sleep(2 * time.Second)
    }
}

The fetchFollowers() method fetches the list of bot followers using twClient. You can specify a maximum cap(Count) on how many latest followers will be allowed to interact with the bot. We will set this to 200. You can change this as per your wish.

We also parse through the followers list to get the ID of our bot followers, and print out their screen-names.

Next, let’s implement the watchFollowers() method.

//Watching bot followers every 60 secs.
//This checks the accounts following
//the bot in every 60 seconds, so that
//any new follower can be added to the
//followers list.
func (b *Bot) watchFollowers() {
    for {
        time.Sleep(60 * time.Second)
        select {
        case <-b.ctx.Done():
        default:
            b.fetchFollowers()
        }
    }
}

As the comments suggest, we call the fetchFollowers() method every 60 seconds to check for new bot followers and update our followers list.

Now, let’s implement the watchTweets() method.

//Function to watch tweets that mentions the bot
func (b *Bot) watchTweets() {
    log.Println("watching tweets")

    /*
        Filter Streams return Tweets that match one
        or more filtering predicates such as Track,
        Follow, and Locations.

        Here we are filtering our tweets with
        "!pin", "!unpin", "!add", "!help" or "<bot-name>" strings
        in tweet body.
    */
    params := &twitter.StreamFilterParams{
        Track: []string{
            PinAction.String(),
            UnpinAction.String(),
            HelpAction.String(),
            AddAction.String(),
            b.Name(),
        },
        StallWarnings: twitter.Bool(true),
    }

    stream, err := b.twClient.Streams.Filter(params)
    if err != nil {
        log.Println(err)
    }

    /*
        Receiving messages of type interface{} isn't very nice,
        it means you'll have to type switch and probably filter
        out message types you don't care about.

        For this,we use Demux, which receives messages and type
        switches them to call functions with typed messages.

        For example, say we're only interested in Tweets.
    */

    demux := twitter.NewSwitchDemux()
    demux.Tweet = func(t *twitter.Tweet) {
        //Processing the tweets
        b.processTweet(t, t)
    }

    //Constantly watching for new filtered tweets
    for {
        select {
        case <-b.ctx.Done():
            return
        case msg := <-stream.Messages:
            //"Handle(msg)" determines the type of a message
            //and calls the corresponding receiver
            //function with the typed message.
            go demux.Handle(msg)
        }
    }
}

func (a Action) String() string {
    return string(a)
}

The watchTweets() method continuously watches for different tweets, by adding a filter on the tweets. We filter for specific strings occurrence in the tweets, so that we don't reply to the every tweet by who mentions the @botHandle.

After filtering the tweets, we process the tweets using processTweet(t, t).

import (
    "github.com/ipfs/go-cid"
    "github.com/ipfs/ipfs-cluster/api"
)

//Process the filetered tweets and handle the tweets according to the
//desired action.
func (b *Bot) processTweet(tweet *twitter.Tweet, srcTweet *twitter.Tweet) {
    if tweet == nil {
        return
    }

    if srcTweet == nil {
        srcTweet = tweet
    }

    // Skip processing our own tweets (written by us)
    // and quotes or retweets we've made (origUser is us)
    // (avoid potential loops)
    if tweet.User.IDStr == b.ID() || srcTweet.User.IDStr == b.ID() {
        return
    }

    action, arguments, urls, err := b.parseTweet(tweet)
    if err != nil {
        b.tweet(err.Error(), tweet, srcTweet, false)
        return
    }

    log.Printf("Parsed: %s, %s, %s\n", action, arguments, urls)

    _, ok := b.followedBy.Load(srcTweet.User.ID)
    if !ok && action.Valid() {
        log.Println("Error: NoFollow")
        b.tweet("Follow me, and try again.", tweet, srcTweet, false)
        return
    }
    if !ok {
        return
    }

    // Process actions
    switch action {
    case PinAction:
        //Pin the Tweet to the IPFS Cluster network
        b.pin(arguments, tweet, srcTweet)
    case UnpinAction:
        //UnPin the Tweet to the IPFS Cluster network
        b.unpin(arguments, tweet, srcTweet)
    case AddAction:
        ////Add the Tweet to the IPFS Cluster network
        b.add(arguments, tweet, srcTweet)
    case HelpAction:
        //Tweet a "help tweet" that demostrates how to use the bot
        b.tweetHelp(tweet, srcTweet)
    default:
        //Need to handle default tweets by adding the assets
        log.Println("no handled action for this tweet")
    }

    // Add any media urls
    if len(urls) > 0 {
        log.Println("adding media: ", urls)
        out := make(chan *api.AddedOutput, 1)
        go func() {
            cids := []cid.Cid{}
            for added := range out {
                log.Printf("added %s\n", added.Cid)
                cids = append(cids, added.Cid)
            }
            if len(cids) > 0 {
                b.tweetAdded(cids, tweet, srcTweet)
            }
        }()
        params := api.DefaultAddParams()
        params.Wrap = true
        params.Name = "Tweet-" + tweet.IDStr
        err := b.clusterClient.Add(context.Background(), urls, params, out)
        if err != nil {
            log.Println(err)
        }
    }

    // If the tweet has retweets, process them as if they were
    // from this user.
    retweets := []*twitter.Tweet{tweet.QuotedStatus, tweet.RetweetedStatus}
    for _, rt := range retweets {
        b.processTweet(rt, srcTweet)
    }
}

The processTweet(tweet, tweet) method processes the filtered tweets and handles the tweets according to the desired action.

First, we check if the tweet is originated from the bot’s account or not. If the tweet originated from the bot’s account then we return, otherwise we will end up in an infinite loop, tweeting our own tweet.

Then we parse the tweets using parseTweet(tweet) to get the action, arguments & urls from the tweet. We will implement the parseTweet(tweet) in a while.

After that we check 2 things. First, we check that the tweets must originate from accounts that follow the bot. Second, we check if the action is a valid action or not.

We can check the validity of the parsed action using the following function.

func (a Action) Valid() bool {
    switch a {
    case PinAction, UnpinAction, AddAction, HelpAction:
        return true
    }
    return false
}

If the tweet is not from someone who follows the bot, then we return.

Then, in the switch statement, we check for action type and perform different actions accordingly. We will implement these different methods(pin, unpin, etc.) in a while.

After that we check for any urls, and use clusterClient to Add the urls to the IPFS Cluster setup, and return the content identifiers (CIDs).

And finally, we handle an edge case in which if the tweet has retweets, then we process them as if they were from the bot follower, and thus calling the processTweet(rt, srcTweet).

Now, let’s implement the parseTweet(tweet) method.

// parseTweet returns Action, arguments, media urls, and error
func (b *Bot) parseTweet(tweet *twitter.Tweet) (Action, string, []string, error) {
    // Extended tweet? let's use the entities from the extended tweet then.
    if tweet.ExtendedTweet != nil {
        tweet.Entities = tweet.ExtendedTweet.Entities
        tweet.ExtendedEntities = tweet.ExtendedTweet.ExtendedEntities
        tweet.FullText = tweet.ExtendedTweet.FullText

    }
    text := tweet.FullText
    if text == "" {
        text = tweet.Text
    }

    log.Println("Parsing:", text)

    // remote our username if they started with it
    text = strings.TrimPrefix(text, b.name)
    var action Action
    var arguments string

    if text == " "+string(HelpAction) {
        return HelpAction, "", []string{}, nil
    }

    // match to see if any action
    matches := actionRegexp.FindAllStringSubmatch(text, -1)
    if len(matches) > 0 {
        firstMatch := matches[0]
        action = Action(firstMatch[1]) // first group match
        arguments = firstMatch[2]      // second group match
    }

    urls := extractMediaURLs(tweet)
    return action, arguments, urls, nil
}

Here, we first handle the ExtendedTweettype, which is a enterprise feature of Twitter, in which you can can have tweets with no word limit.

Then we apply regular expression to our tweets to check for action & arguments. For extracting the urls we use
extractMediaURLs(tweet).

//Extracting MediaURLs from tweets
func extractMediaURLs(tweet *twitter.Tweet) []string {
    var urls []string

    // Grab any media entities from the tweet
    for _, m := range media(tweet.ExtendedEntities) {
        urls = append(urls, extractMediaURL(&m))
    }

    if len(urls) == 0 {
        // If no extended entitites, try with traditional.
        for _, m := range media(tweet.Entities) {
            urls = append(urls, extractMediaURL(&m))
        }
    }
    return urls
}

// takes *Entities or *MediaEntities
func media(ent interface{}) []twitter.MediaEntity {
    if ent == nil {
        return nil
    }

    switch ent.(type) {
    case *twitter.Entities:
        e := ent.(*twitter.Entities)
        if e != nil {
            return e.Media
        }
    case *twitter.ExtendedEntity:
        e := ent.(*twitter.ExtendedEntity)
        if e != nil {
            return e.Media
        }
    }
    return nil
}

Here we loop through the ExtendedEntities and Entities to grab the media entities. Now, we extract the actual MediaURL using the extractMediaURL(MediaEntity) function.

type byBitrate []twitter.VideoVariant

func (vv byBitrate) Len() int           { return len(vv) }
func (vv byBitrate) Swap(i, j int)      { vv[i], vv[j] = vv[j], vv[i] }
func (vv byBitrate) Less(i, j int) bool { return vv[i].Bitrate < vv[j].Bitrate }

//Extracting the highest bitrate MediaURL from MediaEntity
func extractMediaURL(me *twitter.MediaEntity) string {
    switch me.Type {
    case "video", "animated_gif":
        variants := me.VideoInfo.Variants
        sort.Sort(byBitrate(variants))
        // pick video with highest bitrate
        last := variants[len(variants)-1]
        return last.URL
    default:
        return me.MediaURL
    }
}

extractMediaURL(MediaEntity) extracts the MediaURL with the highest bitrate from the available variants of the video or animated_gif URLs.

Now, as we have parsed all the necessary information from the filtered tweets, we can proceed to implement pin, unpin,and add methods.

//Function to pin a CID to IPFS Cluster
func (b *Bot) pin(args string, tweet, srcTweet *twitter.Tweet) {
    log.Println("pin with ", args)
    pinUsage := fmt.Sprintf("Usage: '%s <cid> <name>'", PinAction)

    matches := pinRegexp.FindAllStringSubmatch(args, -1)
    if len(matches) == 0 {
        b.tweet(pinUsage, srcTweet, nil, false)
        return
    }

    firstMatch := matches[0]
    cidStr := firstMatch[1]
    name := firstMatch[2]
    c, err := cid.Decode(cidStr)
    if err != nil {
        b.tweet(pinUsage+". Make sure your CID is valid.", tweet, srcTweet, false)
        return
    }

    _, err = b.clusterClient.Pin(context.Background(), c, api.PinOptions{Name: name})
    if err != nil {
        log.Println(err)
        b.tweet("An error happened pinning. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
        b.die <- struct{}{}
        return
    }
    waitParams := client.StatusFilterParams{
        Cid:       c,
        Local:     false,
        Target:    api.TrackerStatusPinned,
        CheckFreq: 10 * time.Second,
    }
    ctx, cancel := context.WithTimeout(b.ctx, 10*time.Minute)
    defer cancel()
    _, err = client.WaitFor(ctx, b.clusterClient, waitParams)
    if err != nil {
        log.Println(err)
        b.tweet("IPFS Cluster has been pinning this for 10 mins. This is normal for big files. Otherwise, make sure there are providers for it. Don't worry, Cluster will keep at it for a week before giving up.", srcTweet, nil, false)
        return
    }

    b.tweet(fmt.Sprintf("Pinned! Check it out at %s/ipfs/%s", IPFSGateway, cidStr), tweet, srcTweet, true)
}

Here we again apply regular expression on args to validate the tweet text and content identifier(cidStr) and name(name) for the pin to be added to IPFS Cluster setup.

We have added a number of error checks for:

Checking of the CID(cidStr) is valid.
Checking if some unexpected error happened while pinning the CID to IPFS Cluster setup using clusterClient.Pin.
Checking if the pinning process takes more that 10 seconds(timeout), which is sufficient for big files.

In case, everything is fine, the bot tweets the IPFSGateway link for the added pin using the tweet method. We will implement the tweet method in a while.

Now, let’s see the unpin method.

// Function to unpin a CID from IPFS Cluster network
func (b *Bot) unpin(args string, tweet, srcTweet *twitter.Tweet) {
    log.Println("unpin with ", args)
    unpinUsage := fmt.Sprintf("Usage: '%s <cid>'", UnpinAction)

    c, err := cid.Decode(args)
    if err != nil {
        b.tweet(unpinUsage+". Make sure your CID is valid.", tweet, srcTweet, false)
        return
    }

    _, err = b.clusterClient.Unpin(context.Background(), c)
    if err != nil && !strings.Contains(err.Error(), "uncommited to state") {
        log.Println(err)
        b.tweet("An error happened unpinning. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
        b.die <- struct{}{}
        return
    }
    waitParams := client.StatusFilterParams{
        Cid:       c,
        Local:     false,
        Target:    api.TrackerStatusUnpinned,
        CheckFreq: 10 * time.Second,
    }
    ctx, cancel := context.WithTimeout(b.ctx, time.Minute)
    defer cancel()
    _, err = client.WaitFor(ctx, b.clusterClient, waitParams)
    if err != nil {
        log.Println(err)
        b.tweet("IPFS Cluster did not manage to unpin the item, but it's trying...", srcTweet, nil, false)
        return
    }

    b.tweet(fmt.Sprintf("Unpinned %s! :'(", args), tweet, srcTweet, false)
}

In the unpin method we use cid.Decode to get the CID(c) from the args.
Similar to pin, here also we have a few checks in place to handle errors.

Making sure that the CID(c) is valid.
Checking if some unexpected error happened while unpinning the CID from IPFS Cluster setup using clusterClient.Unpin.
Checking if the unpinning process takes more that 10 seconds(timeout).

In case, everything is fine, the bot tweets the acknowledging that the CID has been unpinned.

Now, let’s see the add method.

//Function to add URL to IPFS Cluster network
func (b *Bot) add(arg string, tweet, srcTweet *twitter.Tweet) {
    log.Println("add with ", arg)
    addUsage := fmt.Sprintf("Usage: '%s <http-or-https-url>'")
    url, err := url.Parse(arg)
    if err != nil {
        b.tweet(addUsage+". Make sure you gave a valid url!", srcTweet, nil, false)
        return
    }
    if url.Scheme != "http" && url.Scheme != "https" {
        b.tweet(addUsage+". Not an HTTP(s) url!", srcTweet, nil, false)
        return
    }

    if url.Host == "localhost" || url.Host == "127.0.0.1" || url.Host == "::1" {
        b.tweet("ehem ehem...", srcTweet, nil, false)
        return
    }

    out := make(chan *api.AddedOutput, 1)
    go func() {
        cids := []cid.Cid{}
        for added := range out {
            cids = append(cids, added.Cid)
        }
        if len(cids) > 0 {
            b.tweetAdded(cids, tweet, srcTweet)
        }
    }()

    params := api.DefaultAddParams()
    params.Wrap = true
    params.Name = "Tweet-" + tweet.IDStr
    log.Println([]string{arg})
    err = b.clusterClient.Add(context.Background(), []string{arg}, params, out)
    if err != nil {
        log.Println(err)
        b.tweet("An error happened adding. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
        b.die <- struct{}{}
        return
    }
}

The add method is used to add data via URLs in the tweet to the IPFS Cluster setup. We parse the url using url.Pars(arg). As the above 2 methods, add also has checks to counter catch the errors.

Making sure that the URL(c) is valid.
Checking if the url.Scheme and url.Host are valid.
Checking if there is an unexpected error while adding files to the IPFS Cluster setup.

In case, everything is fine, the bot tweets CID(s) of the file(via the URL) and the folder-wrap(file wrapped in a folder) using tweetAdded.

//Function for Tweeting for the Add Action
func (b *Bot) tweetAdded(cids []cid.Cid, tweet, srcTweet *twitter.Tweet) {
    msg := "Just added this to #IPFS Cluster!\n\n"
    for i, c := range cids {
        if i != len(cids)-1 {
            msg += fmt.Sprintf("• File: %s/ipfs/%s\n", IPFSGateway, c)
        } else { // last
            msg += fmt.Sprintf("• Folder-wrap: %s/ipfs/%s\n", IPFSGateway, c)
        }
    }
    b.tweet(msg, tweet, srcTweet, true)
}

Here is the implementation of the tweetHelp method which tweets how to use the bot.

//Function for Tweeting for the Help Action
func (b *Bot) tweetHelp(tweet, srcTweet *twitter.Tweet) {
    help := fmt.Sprintf(`Hi! Here's what I can do:

!pin <cid> <name>
!unpin <cid>
!add <url-to-single-file>
!help

You can always prepend these commands mentioning me (%s).

Happy pinning!
`, b.name)
    b.tweet(help, srcTweet, nil, false)
}

Now, let’s implement the tweet method which sends tweets quoting or replying to the filtered tweets.

// tweets sends a tweet quoting or replying to the given tweets.
// srcTweet might be nil.
// Otherwise it just posts the message.
func (b *Bot) tweet(msg string, inReplyTo, srcTweet *twitter.Tweet, quote bool) {
    tweetMsg := ""
    params := &twitter.StatusUpdateParams{}
    sameTweets := false

    if inReplyTo == nil {
        tweetMsg = msg
        goto TWEET
    }

    sameTweets = srcTweet == nil || inReplyTo.ID == srcTweet.ID
    params.InReplyToStatusID = inReplyTo.ID

    switch {
    case sameTweets && !quote:
        // @user msg (reply thread)
        tweetMsg = fmt.Sprintf("@%s %s", inReplyTo.User.ScreenName, msg)
    case sameTweets && quote:
        // @user msg <permalink> (quote RT)
        tweetMsg = fmt.Sprintf(".@%s %s %s",
            inReplyTo.User.ScreenName,
            msg,
            permaLink(inReplyTo),
        )
    case !sameTweets && !quote:
        // @user @srcUser msg (reply thread)
        tweetMsg = fmt.Sprintf("@%s @%s %s",
            inReplyTo.User.ScreenName,
            srcTweet.User.ScreenName,
            msg,
        )
    case !sameTweets && quote:
        // @srcuser <replyPermalink> (quote RT mentioning src user)
        tweetMsg = fmt.Sprintf(".@%s %s %s",
            srcTweet.User.ScreenName,
            msg,
            permaLink(inReplyTo),
        )

    }

TWEET:
    log.Println("tweeting:", tweetMsg)
    newTweet, _, err := b.twClient.Statuses.Update(tweetMsg, params)
    if err != nil {
        log.Println(err)
        return
    }
    _ = newTweet
    // if quote { // then retweet my tweet after a minute
    //  go func() {
    //      time.Sleep(time.Minute)
    //      _, _, err := b.twClient.Statuses.Retweet(newTweet.ID, nil)
    //      log.Println("retweeted: ", tweetMsg)
    //      if err != nil {
    //          log.Println(err)
    //          return
    //      }
    //  }()
    // }
    return
}

func permaLink(tweet *twitter.Tweet) string {
    return fmt.Sprintf("https://%s/%s/status/%s", twittercom, tweet.User.ScreenName, tweet.IDStr)
}

Now, finally, let’s implement the main function of our pinbot, which will complete our pinbot code.

func main() {
    //Fetching the optional path from command line
    path := flag.String("config", ConfigFile, "path to config file")
    flag.Parse()

    //Reading the config file
    cfg := readConfig(*path)

    //Creating a new bot
    bot, err := New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    log.Println("Bot created:", bot.Name(), bot.ID())

    // Wait for SIGINT and SIGTERM (HIT CTRL-C)
    ch := make(chan os.Signal)
    signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
    select {
    case sig := <-ch:
        log.Println(sig)
    case <-bot.die:
    }

    bot.Kill()
}

In the main function, we read the config file from the path using the readConfig function. Then using the config file key-values(cfg) we create our bot using the New(cfg) function.

Also, we wait for SIGINT and SIGTERM so that hitting CTRL-C kills out bot.

Final Showdown

Now, it’s time to test our code.

First build the code by executing go build in the directory/folder(twitter-pinbot) where you have saved your main.go file.

This will download all the modules that we have imported. After this is done, you will see an executable file named twitter-pinbot.

Before starting, make sure that both, IPFS dameon and IPFS Cluster daemon are running, that we started above.

Now, you can run the executable file by executing ./twitter-pinbot.

If everything works fine, then you would see a list of your bot followers like this:

Now, we are waiting for your tweets…

If you get stuck in any part or have any queries/doubts, then feel free to
reach us out on our **discord channel.**

Hands-on IPLD Tutorial in Golang: PART 2

vasa — Thu, 16 Jan 2020 14:27:31 +0000

This article was first published on our open-source platform, SimpleAsWater.com. If you are interested in IPFS, Libp2p, Ethereum, Zero-knowledge Proofs, DeFi, CryptoEconomics, IPLD, Multiformats, and other Web 3.0 projects, concepts and interactive tutorials, then be sure to check out SimpleAsWater.

Quick recap from PART-1

Before we explore how to use IPLD for the document store based interface, have you read the PART 1 of this series yet? Here's the link to PART 1 in case you want to catch up!

Getting Started with IPLD: Hands-on IPLD Tutorial in Golang: PART 1

TLDR: In PART 1, we explored the basic concepts of IPLD, how it is used in IPFS. We also scratched the surface by building a key-value based database interface on IPFS using IPLD.

What We Will Learn from PART-2? 🤔

In PART 2 of the series, we will directly dive into the coding and see how to enhance the functionality to build document storage (similar to MongoDB and other document databases).

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

Here is final shown down 😎

Changing the Data Structures 🛠

In contrast to PART 1, our data structure will be a structure which will be further mapped by a string type key. Each document entry we store is of the following format. This is very similar to how you develop schemas:

// SampleStruct defines the benchmark payload
type SampleStruct struct {
    ID     string `json:"ID"`
    Name   string `json:"Name"`
    Salary string `json:"Salary"`
}

As shown above, we have defined SampleStruct, which defines three fields representing an employee's data record. This means, we can contain each employee's ID, Name and corresponding Salary amount in the structure's instances.

Changing the Way We Track Entries 🧐

Further, we need to manage this relationship in mapping by its own ID to facilitate Read or Lookup operations for each employee record. Hence, we update the current mapping as follows:

// Map the struct instance its own ID
DocStoreMap := make(map[string]SampleStruct)

Taking User Input 👨‍💻👩‍💻

Now, we can ask users to enter the values for all the three fields as defined in the SampleStruct. Once we have the value for all the three fields (namely ID, Name and Salary), we can initiate an object/instance based on the definition and assign the received values as follows:

scanner := bufio.NewScanner(os.Stdin)

fmt.Println("Enter the ID of the employee: ")
scanner.Scan()
inputID := scanner.Text()

fmt.Println("Enter the name of the employee: ")
scanner.Scan()
inputName := scanner.Text()

fmt.Println("Enter the salary of the employee: ")
scanner.Scan()
inputSalary := scanner.Text()

// Create a struct instance and assign the input values to the corresponding fields
employeeObject := SampleStruct{ID: inputID, Name: inputName, Salary: inputSalary}

The Document-Storage Magic 🔮✨

Once the object is created and the captured values are assigned, we can map the same object in the mapping by its own ID value. Once the mapping is updated, we can now write it to the Merkle DAG as follows:

// Map the struct instance to the mapping
DocStoreMap[inputID] = employeeObject

// Converting the map into JSON object
entryJSON, err := json.Marshal(DocStoreMap)
if err != nil {
    fmt.Println(err)
}

// Display the marshaled JSON object before sending it to IPFS
jsonStr := string(entryJSON)
fmt.Println("The JSON object of your document entry is:")
fmt.Println(jsonStr)

start := time.Now()
// Dag PUT operation which will return the CID for futher access or pinning etc.
cid, err := sh.DagPut(entryJSON, "json", "cbor")
elapsed := time.Since(start)
if err != nil {
    fmt.Fprintf(os.Stderr, "error: %s", err)
    os.Exit(1)
}

If you are scratching your heading thinking what does a DAG means & what does DagPut function do, then we recommend going through the PART 1 of the series.

We have now successfully written the mapping to the IPLD DAG and a CID may be returned. Yay🎉🎉

A Change in Reading DAGs for Entries

Now, we must update our code to read the document from the mapping. In contrast to PART 1, the read operation will not be returning a mere single-value. It will be returning the whole JSON document as defined in the schema. Hence, we update the getter function by changing the return type as follows:

// GetDocument handles READ operations of a DAG entry by CID, returning the corresponding document
func GetDocument(ref, key string) (out SampleStruct, err error) {
    err = sh.DagGet(ref+"/"+key, &amp;out)
    return
}

Now, let us call this getter/read function in main.go as follows:

// Fetch the details by reading the DAG for key "inputKey"
fmt.Printf("READ: Reading the document details of employee by ID: \"%s\"\n", inputID)
start = time.Now()
document, err := GetDocument(cid, inputID)
elapsed = time.Since(start)
if err != nil {
    fmt.Println(err)
}

fmt.Printf("READ: Salary of employee ID %s is %s\n", string(inputID), string(document.Salary))

As shown above, the GetDocument is passed with the ID value which was earlier entered by the user. Based on the ID value, the corresponding document is returned from the mapping.
Once the document is returned, we can parse it as required and print the Salary of the inquired employee.

Conclusion

We hope you had fun learning how to enhance the code to support document storage. Go crazy and build your own decentralized document database! 🤪

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

And here is the link to the full implementation of the code on GitHub.

About the Authors:

Ganesh Prasad Kumble

Ganesh is an expert in emerging technologies and business strategy. He has co-founded, bootstrapped, mentored several start-ups and initiatives across SaaS, eCommerce, IoT, Blockchain & AI.

He is a contributor to several open-source projects including Ethereum and IPFS. He is also a moderator at the Ethereum Research forum.

Ganesh is currently leading Platform Innovation efforts at Aicumen Technologies Inc. & KIP Foundation, building a general purpose business protocol featuring identity management, third party services, distributed compute and immutable storage.

He is also the author of the Hands-on Artificial Intelligence for Blockchain book.

Vaibhav Saini

Vaibhav Saini is Co-founder of TowardsBlockchain, Dappkit & SimpleAsWater. You can know more about him here.

Hands-on IPLD Tutorial in Golang: PART 1

vasa — Sat, 04 Jan 2020 18:05:15 +0000

In this first post of the hands-on series, we will understand the basic concepts of IPLD followed by coding in Golang to persist key-value entries.

Understanding IPLD
Install Golang
Use go-ipfs-api to write & fetch data via IPLD.
Use IPLD explorer to explore the data that we wrote using the command line.

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

Here is a preview of the final implementation.

You can check out the complete code for the tutorial here.

Understanding IPLD

When you add the photo to IPFS, this is what happens:

When we add something to IPFS, we get something like this:

$ ipfs add cat.png 
added QmQgQUbBeMTnH1j3QWwNw9LkXjpWDJrjyGYfZpnPp8x5Lu cat.png
 717.23 KiB / 717.23 KiB [=============================================] 100.00%

You can see the final hash here:

QmQgQUbBeMTnH1j3QWwNw9LkXjpWDJrjyGYfZpnPp8x5Lu

But we don’t see anything related to the 2 steps(Raw and Digest). This all happens under the hood.

When we added the image, we converted the image into the Raw data which the computer can understand. Now, to make it content-addressable we have to come up with a method by which we can convert this image data to a label that uniquely identifies it’s content.

This is where hash functions come into play.

Hash functions take the data(any data from texts, photos, whole bible, etc.) as input and give us an output(Digest) which is unique with respect to its output. If we change even a pixel in this image then the output will be different. This shows its tamper-proof property, hence making IPFS a Self-certifying File System. So if you transfer this image to anybody, he/she can check that if the photo received has been tampered with or not.

Also, you cannot tell what was the input(in this case, a cat photo), but just seeing its output(the Digest). So, this also ensures a great amount of security.

Now we pass the Raw image data into SHA256 hash function and get the unique Digest. Now, we need to convert this Digest into a CID(Content Identifier). This CID is what IPFS will search for when we try to get back the image. For doing this, IPFS uses something called Multihash.

To understand the significance of Multihash, consider this situation.

You stored this image on the internet and you have its CID, which you can give to anybody who wants to get this image. Now, what if you discover in future that SHA256 is broken(this would mean that this process is NO more tamper-proof and secure) and you want to use SHA3(to ensure tamper-proofing and security) instead? This would mean changing the whole process of converting your photo to CID, and the previous CIDs will be useless…

This above problem may seem a small issue in this context, but you should know that these hash functions secure TRILLIONS of dollars. All the Banks, National security agencies, etc. use these hash functions to operate securely. Even the green lock that you see beside the address of every site on the browser will not work without it.

In order to solve this problem, IPFS uses Multihash. Multihash allows us to define hashes that are self-defining. So, we can have multiple versions of CIDs, according to the hash function used. We will talk more about it below.

Well, now we have added our photo to IPFS, but this was not the whole story. What is actually happening is something like this:

If the files are bigger than 256 kB, then they are broken down into smaller parts, so that all the parts are equal or smaller than 256 kb. We can see the chunks of our photo using this command:

$ ipfs object get QmQgQUbBeMTnH1j3QWwNw9LkXjpWDJrjyGYfZpnPp8x5Lu

This gives us 3 chunks, each of which is smaller than 256kb. Each of these chunks is first converted into a digest and then into CIDs.

{
    "Links": [
        {
            "Name": "",
            "Hash": "QmYN9f4cRGPReJDSi3YoFTt5eTVS2Jo9ePN3wH3TfgbB8u",
            "Size": 262158
        },
        {
            "Name": "",
            "Hash": "QmTJ1rwQQ7FC4HiwmxS1jFe2eJeb6kyxgRWKGyHjf7nYMN",
            "Size": 262158
        },
        {
            "Name": "",
            "Hash": "QmSEuztdUaJNLGhf3Hrpd9f8eHXftusY8QCbqUbzGv7LNX",
            "Size": 210174
        }
    ],
    "Data": "\u0008\u0002\u0018��, ��\u0010 ��\u0010 ��\u000c"
}

IPFS uses IPLD (IPLD uses Merkle DAG, or directed acyclic graph) for managing all the chunks and linking it to the base CID.

IPLD (objects) consist of 2 components:

Data — a blob of unstructured binary data of size < 256 kB.
Links — array of Link structures. These are links to other IPFS objects.

Every IPLD Link(in our case the 15 links that we got above) has 3 parts:

Name — name of the Link
Hash — the hash of the linked IPFS object
Size — the cumulative size of linked IPFS object, including following its links

IPLD is built on the ideas of Linked Data, which is actually something that folks in the decentralized web community have been talking about for quite some time. It’s something Tim Berners-Lee has been working on for ages, and his new company, Solid, is building a business around it.

Using IPLD has also other benefits. To see this, let’s create a folder named photos and add 2 photos into it (the cat pic and a copy of the same pic).

As you can see both the photos have the same hash(which proves that I haven’t changed anything in the copy of the image). This adds De-duplication property to IPFS. So even if your friend adds the same cat photo to IPFS, he will not duplicate the image. This saves a lot of storage space.

Here is a talk by Juan Benet explaining the significance and features of IPLD that we just discussed.

Now, as we have understood what is IPLD, how it works and its significance, let’s get our hands dirty!

Playing With IPLD

In IPFS, IPLD helps to structure and link all the data chunks/objects. So, as we saw above, IPLD was responsible for organizing all the data chunks that constituted the image of the kitty🐱.

In this part, we will use go-ipfs-api to write and fetch data via IPLD. Let's get started!

Install Golang

We recommend you to install the latest version of Golang from the official website](https://golang.org/doc/install). There are instructions available for installing Golang on Windows, Linux, FreeBSD & MacOS.

Writing to IPFS via IPLD

The purpose of the code is to get access to some of the deeper abstractions of IPFS that can allow developers to store information apart from files. This was possible by operating directly on the Merkle DAGs. Although versions of similar implementations are available in Javascript, we wanted to implement something in Golang.

Coding the IPLD operations block-by-block

The code primarily depends upon the go package called go-ipfs-api. It is maintained by IPFS and offers wrappers to many core HTTP APIs.

Our code is a single file that defines every key-value through a simple mapping:

// Mapping-based structure to record key-value information
keyValueMap := make(map[string]interface{})

Followed by the definition of key-value structure, we read the user inputs. Once we read the user inputs of key and value, we set them in the mapping using the following code:

scanner := bufio.NewScanner(os.Stdin)

fmt.Println("Enter value for the key field: ")
scanner.Scan()
inputKey := scanner.Text()

fmt.Println("Enter value for value field: ")
scanner.Scan()
inputValue := scanner.Text()

// Setting the key and values in the mapping
m[inputKey] = inputValue

The mapping set with user inputs will now be converted into a JSON object and printed on the console with the following code:

// Converting into JSON object
entryJSON, err := json.Marshal(m)
if err != nil {
    fmt.Println(err)
}

// Display the marshaled JSON object before sending it to IPFS
jsonStr := string(entryJSON)
fmt.Println("The JSON object of your key-value entry is:")
fmt.Println(jsonStr)

Once the JSON object is created and marshaled, we now add this to the DAG by calling the DagPut() function:

// Dag PUT operation which will return the CID for futher access or pinning etc.
cid, err := sh.DagPut(entryJSON, "json", "cbor")
if err != nil {
    fmt.Fprintf(os.Stderr, "error: %s", err)
    os.Exit(1)
}
fmt.Println("------\nOUTPUT\n------")
fmt.Printf("WRITE: Successfully added %sHere's the IPLD Explorer link: https://explore.ipld.io/#/explore/%s \n", string(cid+"\n"), string(cid+"\n"))

Once the JSON object is written, the DagPut() returns the CID.

We can now access the IPLD entry by passing along the returned CID to the GetDag() function. Once we receive the entry, we also print it on the console using the following code:

// Fetch the details by reading the DAG for key "inputKey"
fmt.Printf("READ: Value for key \"%s\" is: ", inputKey)
res, err := GetDag(cid, inputKey)
if err != nil {
    fmt.Println(err)
}
fmt.Println(res)

The full implementation can be found here.

Here is how our code looks in action😎

Conclusion

In summary, we have covered the basic concepts of hashing, IPFS and how IPLD is used. Further, we have also provided a hands-on code demonstration in Golang to effectively building a key-value based data interface using the go-ipfs-api package.

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

In PART 2 of the series, we will be introducing how to extend this code for enacting the storage of docstore entries.

Hands-on IPLD Tutorial in Golang: PART 2

About the authors

Ganesh Prasad Kumble

Ganesh is an expert in emerging technologies and business strategy. He has co-founded, bootstrapped, mentored several start-ups and initiatives across SaaS, eCommerce, IoT, Blockchain & AI.

He is a contributor to several open-source projects including Ethereum and IPFS. He is also a moderator at the Ethereum Research forum.

He is currently leading Platform Innovation efforts at Aicumen Technologies Inc. & KIP Foundation, building a general purpose business protocol featuring identity management, third party services, distributed compute and immutable storage.

Ganesh is the author of the Hands-On Artificial Intelligence for Blockchain book.

You can know more about him here.

Vaibhav Saini

Vaibhav is the Co-founder of TowardsBlockchain, SimpleAsWater & Dappkit. You can know more about him here.

Understanding IPFS in Depth(2/6): What is InterPlanetary Linked Data(IPLD)?

vasa — Tue, 31 Dec 2019 17:45:39 +0000

This post is a continuation(part 2) in a new “Understanding IPFS in Depth” series which will help anybody to understand the underlying concepts of IPFS. If you want an overview of what is IPFS and how it works, then you should check out the first part too 😊

Understanding IPFS in Depth(1/6): A Beginner to Advanced Guide

In our first part of the series we briefly discussed IPLD. We saw that IPLD deals with “defining the data” in IPFS(well, not just in IPFS). In this part, we will dive fully into IPLD and discuss:

The Significance of IPLD: What is the philosophy behind it, Why do we need it and Where does it fit into IPFS?
How does IPLD Work?: An Explanation of its specification and how does it coordinate with other components of IPFS?
Playing with IPLD: Well, It’s always fun to play 😊

If you like high-tech Web3 concepts like IPLD explained in simple words with interactive tutorials, then check out SimpleAsWater.com.

I hope you learn a lot about IPFS from this series. Let’s start!

The Significance of IPLD

IPLD is not just a part of the IPFS project, but a separate project in itself. To understand its significance in the decentralized world we have to understand the concept of Linked Data: The Semantic Web.

What is Linked Data & Why do we Need it?

The Semantic Web or Linked Data is a term coined by Sir Tim Berners Lee in a seminal article published in Scientific American in 2001. Berners Lee articulated a vision of a World Wide Web of data that machines could process independently of humans, enabling a host of new services transforming our everyday lives. While the paper’s vision of most web pages containing structured data that could be analyzed and acted upon by software agents has NOT materialized, the Semantic Web has emerged as a platform of increasing importance for data interchange and integration through the growing community implementing data sharing using international semantic web standards, called Linked Data.

There are many current examples of the use of semantic web technologies and Linked Data to share valuable structured information in a flexible and extensible manner across the web. Semantic Web technologies are used extensively in the life sciences to facilitate drug discovery by finding paths across multiple datasets showing associations between drugs and side effects via genes linked to each. The New York Times has published its vocabulary of approximately 10,000 subject headings developed over 150 years as Linked Data and will expand coverage to approximately 30,000 topic tags; they encourage the development of services consuming these vocabularies and linking them with other online resources. The British Broadcasting Corporation uses Linked Data to make content more findable by search engines and more linkable through social media; to add additional context from supplemental resources in domains like music or sports, and to propagate linkages and editorial annotations beyond their original entry target to bring relevant information forward in additional contexts. The home page of the United States data.gov site states, “As the web of linked documents evolves to include the Web of linked data, we’re working to maximize the potential of Semantic Web technologies to realize the promise of Linked Open Government Data.” And also all the social media websites use Linked Data to create a web of People to make their platform as engaging as possible.

So we do have some Linked Data but we still have a long way to go in order to harness the true power of Linked Data.

Now imagine if you could refer your latest commits in a git branch to a bitcoin transaction to timestamp your work. So, by linking your git commit, you can view the commit from your blockchain explorer. Or, if you could link your Ethereum contract to media on IPFS, perhaps modifying it and tracking its changes on each function execution.

All this is possible using IPLD.

IPLD is the data model of the content-addressable web(as discussed in part1).

It allows us to treat all hash-linked data structures as subsets of a unified information space, unifying all data models that link data with hashes as instances of IPLD.

Or in other words, IPLD is a set of standards and implementations for creating decentralized data-structures that are universally addressable and linkable. These structures allow us to do for data what URLs and links did for HTML web pages.

Content addressing through hashes has become a widely-used means of connecting data in distributed systems, from the blockchains that run your favorite cryptocurrencies, to the commits that back your code, to the web’s content at large. Yet, whilst all of these tools rely on some common primitives, their specific underlying data structures are NOT interoperable(as right now I can’t connect my git commit to a blockchain transaction).

IPLD is a single namespace for all hash-inspired protocols. Through IPLD, links can be traversed across protocols, allowing you to explore data regardless of the underlying protocol.

Before diving deeper into IPLD, Let’s see the properties of IPLD.

Properties of IPLD

The sky’s the limit as IPLD allows you to work across protocol boundaries. The point is that IPLD provides libraries that make the underlying data interoperable across tools and across protocols by default.

A canonical data model

A self-contained descriptive model that uniquely identifies any hash-based data structure and ensures the same logical object always maps to the exact same sequence of bits.

Protocol independent resolution

IPLD brings isolated systems together(like connecting Bitcoin, Ethereum and git), making integration with existing protocols simple.

Upgradeable

With Multiformats(we will dive more into this in part 4) support, IPLD is easily upgradeable and will grow with your favorite protocols.

Operates across formats

Express your IPLD objects in various serializable formats like JSON, CBOR, YAML, XML and many more, making IPLD a cinch to use with any framework.

Backward compatible

Non-intrusive resolvers make IPLD easy to integrate within your existing work.

A namespace for all protocols

IPLD allows you to explore data across protocols seamlessly, binding hash-based data structures together through a common namespace.

Now, let’s dive deeper into the IPLD.

Diving Deeper into IPLD Specs

IPLD is not a single specification, it is a set of specifications.

The goal of this stack is to enable decentralized data-structures which in turn will enable more decentralized applications.

Many of the specifications in this stack are inter-dependent.

These diagrams show a high level of project specification of IPLD. Even if you don’t understand it fully it’s totally OK.

To learn more about IPLD, here is a great talk by Juan Benet.

OK. Enough of theoretical stuff. Let’s dive into the most fun part of this post 😊

In IPFS, IPLD helps to structure and link all the data chunks/objects. So, as we saw in part 1, IPLD was responsible for organizing all the data chunks that constituted the image of the kitty🐱.

Playing with IPLD

In IPFS, IPLD helps to structure and link all the data chunks/objects. So, as we saw in part 1, IPLD was responsible for organizing all the data chunks that constituted the image of the kitty🐱.

In this part, we will create a medium.com like publication system, and link the tags, articles, and authors using IPLD. This will help you to get a more intuitive understanding of IPLD. You can also find the complete tutorial on Github.

Let’s get started!

Before creating the publication system, we’ll be exploring the IPFS DAG API, which lets us store data objects in IPFS in IPLD format. (You can store more exciting things in IPFS, like your favorite cat GIF, but we will stick to simpler things for now.)

If you are not familiar with Merkle tries and DAGs then head over here. If you have an understanding of what these terms mean then continue…

Create a folder name ipld-blogs. Run npm init and press Enter for all the questions.

Now install the dependencies using:

npm install ipfs-http-client cids --save

After installing the module your project structure will look like this:

Creating an IPLD format node

You can create a new node by passing a data object into the ipfs.dag.put method, which returns a Content Identifier (CID) for the newly created node.

ipfs.dag.put({name: ‘vasa’})

A CID is an address for a block of data in IPFS that is derived from its content. Every time someone puts the same {name: 'vasa'} data into IPFS, they'll get back an identical CID to the one you got. If they put in {name: 'vAsa'} instead, the CID will be different.

Paste this code in tut.js and run node tut.js

//Initiate ipfs and CID instance
const ipfsClient = require('ipfs-http-client');
const CID = require('cids');

//Connecting ipfs instance to infura node. You can also use your local node.
const ipfs = new ipfsClient({ host: 'ipfs.infura.io', port: '5001', protocol: 'https' });

/*
Creating an IPLD format node:
ipfs.dag.put(dagNode, [options], [callback])
For more information see: 
https://github.com/ipfs/interface-js-ipfs-core/blob/master/SPEC/DAG.md#ipfsdagputdagnode-options-callback
*/

ipfs.dag.put({name: "vasa"}, { format: 'dag-cbor', hashAlg: 'sha2-256' }, (err, cid)=>{
  if(err){
      console.log("ERR\n", err);
  }

  //featching multihash buffer from cid object.
  const multihash = cid.multihash;

  //passing multihash buffer to CID object to convert multihash to a readable format   
  const cids = new CID(1, 'dag-cbor', multihash);

  //Printing out the cid in a readable format
  console.log(cids.toBaseEncodedString());
  //zdpuAujL3noEMamveLPQWJPY6CYZHhHoskYQaZBvRbAfVwR8S
});

You will get this CID zdpuAujL3noEMamveLPQWJPY6CYZHhHoskYQaZBvRbAfVwR8S. We have successfully created an IPLD format node.

Connecting IPLD objects

One important feature of Directed Acyclic Graphs (DAGs) is the ability to link them together.

The way you express links in the ipfs DAG store is with the CID of another node.

For example, if we wanted one node to have a link called “foo” pointed to another CID instance previously saved as barCid, it might look like:

{
  foo: barCid
}

When we give a field a name and make its value a link to a CID, we call this a named link.

Below is the code showing how to create a named link.


//Initiate ipfs and CID instance
const ipfsClient = require('ipfs-http-client');
const CID = require('cids');

//Connecting ipfs instance to infura node. You can also use your local node.
const ipfs = new ipfsClient({ host: 'ipfs.infura.io', port: '5001', protocol: 'https' });

/*
Creating an IPLD format node:
ipfs.dag.put(dagNode, [options], [callback])
For more information see: 
https://github.com/ipfs/interface-js-ipfs-core/blob/master/SPEC/DAG.md#ipfsdagputdagnode-options-callback
*/
async function linkNodes(){
    let vasa = await ipfs.dag.put({name: 'vasa'});

    //Linking secondNode to vasa using named link.
    let secondNode = await ipfs.dag.put({linkToVasa: vasa});
}

linkNodes();

Reading Nested data using Links

You can read data from deeply nested objects using path queries.

ipfs.dag.get allows queries using IPFS paths. These queries return an object containing the value of the query and any remaining path that was unresolved.

The cool thing about this API is that it can also traverse through links. Here is an example of how you can read nested data using links.

//Initiate ipfs and CID instance
const ipfsClient = require('ipfs-http-client');
const CID = require('cids');

//Connecting ipfs instance to infura node. You can also use your local node.
const ipfs = new ipfsClient({ host: 'ipfs.infura.io', port: '5001', protocol: 'https' });

function errOrLog(err, result) {
    if (err) {
      console.error('error: ' + err)
    } else {
      console.log(result)
    }
}

async function createAndFeatchNodes(){
    let vasa = await ipfs.dag.put({name: 'vasa'});

    //Linking secondNode to vasa using named link.
    let secondNode = await ipfs.dag.put({
        publication: {
            authors: {
                authorName: vasa
            }
        }
    });

    //featching multihash buffer from cid object.
    const multihash = secondNode.multihash;

    //passing multihash buffer to CID object to convert multihash to a readable format   
    const cids = new CID(1, 'dag-cbor', multihash);

    //Featching the value using links
    ipfs.dag.get(cids.toBaseEncodedString()+'/publication/authors/authorName/name', errOrLog);
    /* prints { value: 'vasa', remainderPath: '' } */
}
createAndFeatchNodes();

You can also explore your IPLD nodes using this cool IPLD explorer. Like, if I want to see this CID: zdpuAujL3noEMamveLPQWJPY6CYZHhHoskYQaZBvRbAfVwR8S, I will go to this link:

https://explore.ipld.io/#/explore/zdpuAujL3noEMamveLPQWJPY6CYZHhHoskYQaZBvRbAfVwR8S

Now, as we have explored IPFS DAG API we are ready to work with IPLD and create our publication system.

Creating a Publication System

We will create a simple blogging application. This blogging application can:

Add a new Author IPLD object. An author will have 2 fields: name and profile(a tag line for your profile).
Create a new Post IPLD object. A post will have 4 fields: author, content, tags and publication date-time.
Read a Post using post CID.

Below is the code implementation for the above goals.

/*
PUBLICATION SYSTEM
Adding new Author
An author will have
-> name
-> profile
Creating A Blog
A Blog will have a:
-> author
-> content
-> tags
-> timeOfPublish
List all Blogs for an author
Read a Blog
*/

//Initiate ipfs and CID instance
const ipfsClient = require('ipfs-http-client');
const CID = require('cids');

//Connecting ipfs instance to infura node. You can also use your local node.
const ipfs = new ipfsClient({ host: 'ipfs.infura.io', port: '5001', protocol: 'https' });

//Create an Author
async function addNewAuthor(name) {
    //creating blog author object
    var newAuthor = await ipfs.dag.put({
        name: name,
        profile: "Entrepreneur | Co-founder/Developer @TowardsBlockChain, an MIT CIC incubated startup | Speaker | https://vaibhavsaini.com"
    });

    console.log("Added new Author "+name+": " + newAuthor);

    return newAuthor;
}

//Creating a Blog
async function createBlog(author, content, tags) {
    //creating blog object
    var post = await ipfs.dag.put({
        author: author,
        content: content,
        tags: tags,
        timeOfPublish: Date()
    });

    console.log("Published a new Post by " + author + ": " + post);

    return post;
}

//Read a blog
async function readBlog(postCID) {
    ipfs.dag.get(postCID + "", (err, post) => {
        if (err) {
          console.error('Error while reading post: ' + err)
        } else {
            console.log("Post Details\n", post);
            ipfs.dag.get(postCID + "/author", (err, author) => {
                if (err) {
                  console.error('Error while reading post author: ' + err)
                } else {
                  console.log("Post Author Details\n", author);
                }
              });
        }
    });
}


function startPublication() {
    addNewAuthor("vasa").then(
      (newAuthor) => {
        createBlog(newAuthor,"my first post", ["ipfs","ipld","vasa","towardsblockchain"]).then(
          (postCID) => readBlog(postCID))
      });
}

startPublication();

On running this code and it will first create an author via addNewAuthor, which will return the authors CID. Then this CID will be passed to createBlog function which will return the postCID. This postCID will be used by readBlog function to fetch the post details.

You can create more complex applications using IPLD…

Ok. that’s it for this part. If you have any question, then you can shoot them in the comments.

I hope you have learned a lot of thing from this post. In the next post, we will dive into the naming System of the distributed web, IPNS. So stay tuned…

This article was first posted on SimpleAsWater.com

Building a Chat Application using Libp2p

vasa — Sat, 28 Dec 2019 21:10:54 +0000

Around the 1960s, the early research on ARPANET(predecessor of the internet)
began around the question that

How do you build a network which is so resilient, so that it could even survive
a NUCLEAR WAR?

The reason that this topic was of such interest in the 1960s was that most of
the communication infrastructure at that time looked like this:

These were banks of human operators connecting wires to facilitate country-wide
communication. This was a highly centralized system, which could be destroyed
easily in a nuclear war.

Fast-forward to today’s world, most of the internet companies today have
centralized servers, most of which are hosted in the data centers outside the
cities. The internet we see today is still somehow full of services similar to
the bank of human operators which are highly centralized and fragile(well, we
aren’t in a constant threat of a nuclear war today, but any technical or natural
disaster can cause great damage).

But, Why do we need to rethink about networking in 2019?

Today, we are dependent on the internet more than ever. We are dependent on it
for life and death situations. But the current internet, as we see it today is
quite fragile and has several design problems. Most of these problems stem from
the location-addressing.

We can go on and on about Why do we need
Libp2p and How it
works.

But, we are here to BUILD STUFF.

So, let’s get right into it.

In this series, we are going to cover a number of posts, which go into the
practical, or even wild applications of peer-to-peer systems using Libp2p. We
will build and also break a lot of stuff.

You can suggest any topic for future tutorials related to Blockchains and in
general Web 3.0.

This is the first in a series of tutorials on working with libp2p’s javascript
implementation, js-libp2p.

Building Chat Application using Libp2p

After going through this tutorial, you will be able to:

Install node.js

Working with js-libp2p requires node.js for development.
If you haven’t already, install node using whatever package manager you prefer
or using the official installer.

We recommend using the latest stable version of node, but anything fairly recent
should work fine. If you want to see how low you can go, the current version
requirements can always be found at the js-libp2p project
page.

Create an empty project

We need a place to put our work, so open a terminal to make a new directory for
your project somewhere and set it up as an npm project:

# create a directory for the project and `cd` into it
$ mkdir -p hello-libp2p/src
$ cd hello-libp2p

# make it a git repository
$ git init .
Initialized empty Git repository in /home/vasa/Desktop/simpleaswater/libp2p/.git/

# make it an npm project
$ npm init -y

Side note: throughout this tutorial, we use the $ character to indicate your
terminal’s shell prompt. When following along, don’t type the $ character,
or you’ll get some weird errors.

Build a libp2p bundle

As we learned in what is Libp2p
post, libp2p is a very modular framework, which allows javascript devs to target
different runtime environments and opt-in to various features by including a
custom selection of modules.

In easy words, let’s suppose you are at a hardware store and want to buy some
tools for your toolbox. Now, you see all the available wrenches in the store.

For most of us, we only want a few sizes of wrenches. So, we pick only the ones
that we want.

Similarly, when we build a networking stack(toolbox) for our application, we
only need a few protocols & modules(wrenches). So, we should be able to select
what protocols and modules we want to use and use them independently.

Libp2p allows you to do the same for networking, as the hardware store does for
the hardware tools, i.e. allowing you to select and use only the tools that you
want to use.

Now, as everyone needs a different set of wrenches, similarily, every
application needs a different “bundle” with just the modules the application
needs.

You can even make more than one bundle if you want to target multiple javascript
runtimes with different features. For example, the IPFS project uses two libp2p
bundles, one for
node.js
and one for the
browser.

Since, we’re here to learn how libp2p works, we’re going to start from scratch
and define our own bundle. We’ll start with a very simple bundle and add
features as we need them.

First, install the libp2p dependency. We’ll also need at least one transport
module, so we’ll pull in libp2p-tcp as well, and the
@nodeutils/defaults-deep helper which we'll use when building the bundle.

$ npm install libp2p@^0.26.2 libp2p-tcp@^0.13.2 @nodeutils/defaults-deep@^1.1.0 --save

In a production application, it may make sense to create a separate npm module
for your bundle, which will give you one place to manage the libp2p
dependencies for all your javascript projects. In that case, you should not
depend on libp2p directly in your application. Instead, you’d depend on your
bundle, which would, in turn, depend on libp2p and whatever modules (transports,
etc) you might need.

For this tutorial, our bundle will just be a javascript file in our application
source.

Make a file called src/libp2p_bundle.js with the following content:

const Libp2p = require('libp2p')
const TCP = require('libp2p-tcp')
const defaultsDeep = require('@nodeutils/defaults-deep')
const DEFAULT_OPTS = {
  modules: {
    transport: [
      TCP
    ]
  }
}
class P2PNode extends Libp2p {
  constructor (opts) {
    super(defaultsDeep(opts, DEFAULT_OPTS))
  }
}
module.exports = P2PNode

The libp2p module exports a libp2p.Node class which we extend into our own
P2PNode class.

Right now our class just adds the libp2p-tcp transport module to the default
constructor options of the base class. As we go, we'll extend our bundle to
include more transports and configure other aspects of the libp2p stack.

Create an instance of a libp2p node

As we know, libp2p was born while working InterPlanetary File System project, it
makes sense to make our libp2p nodes InterPlanetary. Let’s make our first node,
moon.js.

Using the bundle we defined above, we can create a new P2PNode instance.

To do so, create a file called src/moon.js and make it look like this:

'use strict'
/* eslint-disable no-console */
const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
PeerId.createFromJSON(require('./ids/moonId'), (err, peerId) => {
    if (err) {
        throw err
    }
const peerInfo = new PeerInfo(peerId)
    peerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    const nodeListener = new Node({ peerInfo })
})

We also have to create a JSON file containing the peerId of our moon peer.
This will help other peers find our moon peer.

Create a file named src/ids/moonId.json

$ mkdir ids
$ touch moonId.json; touch earthId.json

Now, add the peerId in moonId.json

{
    "id": "QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm",
    "privKey": "CAASqAkwggSkAgEAAoIBAQDLZZcGcbe4urMBVlcHgN0fpBymY+xcr14ewvamG70QZODJ1h9sljlExZ7byLiqRB3SjGbfpZ1FweznwNxWtWpjHkQjTVXeoM4EEgDSNO/Cg7KNlU0EJvgPJXeEPycAZX9qASbVJ6EECQ40VR/7+SuSqsdL1hrmG1phpIju+D64gLyWpw9WEALfzMpH5I/KvdYDW3N4g6zOD2mZNp5y1gHeXINHWzMF596O72/6cxwyiXV1eJ000k1NVnUyrPjXtqWdVLRk5IU1LFpoQoXZU5X1hKj1a2qt/lZfH5eOrF/ramHcwhrYYw1txf8JHXWO/bbNnyemTHAvutZpTNrsWATfAgMBAAECggEAQj0obPnVyjxLFZFnsFLgMHDCv9Fk5V5bOYtmxfvcm50us6ye+T8HEYWGUa9RrGmYiLweuJD34gLgwyzE1RwptHPj3tdNsr4NubefOtXwixlWqdNIjKSgPlaGULQ8YF2tm/kaC2rnfifwz0w1qVqhPReO5fypL+0ShyANVD3WN0Fo2ugzrniCXHUpR2sHXSg6K+2+qWdveyjNWog34b7CgpV73Ln96BWae6ElU8PR5AWdMnRaA9ucA+/HWWJIWB3Fb4+6uwlxhu2L50Ckq1gwYZCtGw63q5L4CglmXMfIKnQAuEzazq9T4YxEkp+XDnVZAOgnQGUBYpetlgMmkkh9qQKBgQDvsEs0ThzFLgnhtC2Jy//ZOrOvIAKAZZf/mS08AqWH3L0/Rjm8ZYbLsRcoWU78sl8UFFwAQhMRDBP9G+RPojWVahBL/B7emdKKnFR1NfwKjFdDVaoX5uNvZEKSl9UubbC4WZJ65u/cd5jEnj+w3ir9G8n+P1gp/0yBz02nZXFgSwKBgQDZPQr4HBxZL7Kx7D49ormIlB7CCn2i7mT11Cppn5ifUTrp7DbFJ2t9e8UNk6tgvbENgCKXvXWsmflSo9gmMxeEOD40AgAkO8Pn2R4OYhrwd89dECiKM34HrVNBzGoB5+YsAno6zGvOzLKbNwMG++2iuNXqXTk4uV9GcI8OnU5ZPQKBgCZUGrKSiyc85XeiSGXwqUkjifhHNh8yH8xPwlwGUFIZimnD4RevZI7OEtXw8iCWpX2gg9XGuyXOuKORAkF5vvfVriV4e7c9Ad4Igbj8mQFWz92EpV6NHXGCpuKqRPzXrZrNOA9PPqwSs+s9IxI1dMpk1zhBCOguWx2m+NP79NVhAoGBAI6WSoTfrpu7ewbdkVzTWgQTdLzYNe6jmxDf2ZbKclrf7lNr/+cYIK2Ud5qZunsdBwFdgVcnu/02czeS42TvVBgs8mcgiQc/Uy7yi4/VROlhOnJTEMjlU2umkGc3zLzDgYiRd7jwRDLQmMrYKNyEr02HFKFn3w8kXSzW5I8rISnhAoGBANhchHVtJd3VMYvxNcQb909FiwTnT9kl9pkjhwivx+f8/K8pDfYCjYSBYCfPTM5Pskv5dXzOdnNuCj6Y2H/9m2SsObukBwF0z5Qijgu1DsxvADVIKZ4rzrGb4uSEmM6200qjJ/9U98fVM7rvOraakrhcf9gRwuspguJQnSO9cLj6",
    "pubKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLZZcGcbe4urMBVlcHgN0fpBymY+xcr14ewvamG70QZODJ1h9sljlExZ7byLiqRB3SjGbfpZ1FweznwNxWtWpjHkQjTVXeoM4EEgDSNO/Cg7KNlU0EJvgPJXeEPycAZX9qASbVJ6EECQ40VR/7+SuSqsdL1hrmG1phpIju+D64gLyWpw9WEALfzMpH5I/KvdYDW3N4g6zOD2mZNp5y1gHeXINHWzMF596O72/6cxwyiXV1eJ000k1NVnUyrPjXtqWdVLRk5IU1LFpoQoXZU5X1hKj1a2qt/lZfH5eOrF/ramHcwhrYYw1txf8JHXWO/bbNnyemTHAvutZpTNrsWATfAgMBAAE="
}

Let’s see what we did above.

In moon.js we start out by importing a few modules. Apart from the P2PNode
bundle we defined earlier, peer-id, which contains a
PeerId associated with a
peer, and peer-info, which provides a libp2p Peer abstraction.

The constructor for our bundle(libp2p_bundle.js) requires a peerInfo
argument. This can either be generated on-the-fly or loaded from a byte buffer
or JSON object. Here, we're generating a new PeerInfo object for our peer
using ids/moonId.json. This will generate a new PeerId containing the
cryptographic key pair that we added in moonId.json.

Once we have our peerId, we next create a
multiaddress for
/ip4/127.0.0.1/tcp/10333, which is the localhost IPv4
address on the TCP port 10333.

Adding the new multiaddr to our peerInfo object will cause our node to try to
listen to that address when the node starts.

Next, we create our peer, passing in the peerInfo constructor option. That's
it for setting up our moon node.

Start the node and listen for connections

Before starting our nodes, let’s add some colors and emojis to our command line
using chalk and node-emoji.

$ npm i chalk node-emoji --save-dev

Now, let’s start our moon node. To do that, replace moon.js with following
code:

'use strict'
/* eslint-disable no-console */
const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
const chalk = require('chalk');
const emoji = require('node-emoji')
PeerId.createFromJSON(require('./ids/moonId'), (err, peerId) => {
    if (err) {
        throw err
    }
    const peerInfo = new PeerInfo(peerId)
    peerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    const nodeListener = new Node({ peerInfo })
nodeListener.start((err) => {
        if (err) {
            throw err
        }
console.log(emoji.get('moon'), chalk.blue(' Moon ready '),
            emoji.get('headphones'), chalk.blue(' Listening on: '));
peerInfo.multiaddrs.forEach((ma) => {
            console.log(ma.toString() + '/p2p/' + peerId.toB58String())
        })
console.log('\n' + emoji.get('moon'), chalk.blue(' Moon trying to connect with Earth '),
            emoji.get('large_blue_circle'));
    })
})

Now, try to run the moon.js, using node moon.js. You will see something like
below.

Now, we have started our moon node and listening for the earth node. So, we need
another node, earth.js. But before building our earth node, we need to add one
final thing to make our moon node work.

In case you are stuck somewhere, let us know here.

Add multiplexing and encryption

We can now start a node and listen for connections, but we can’t really do
anything yet. This is because we’re missing a key libp2p component called a
stream multiplexer,
which lets us interleave multiple independent streams of communication across
one network connection.

But, what is multiplexing?

To understand multiplexing, let’s take an example of your TV cable or WiFi
router. You can see multiple channels, which all comes through a single cable
attached from your setup box. Also, multiple people can connect to the same WiFi
router and watch videos and read an article all at the same time…Ever wondered
how you can do multiple things using a single wire or router?

The answer is multiplexing.

Multiplexing (or muxing) is a way of sending multiple signals or streams of
information over a communications link at the same time in the form of a single,
complex signal; the receiver recovers the separate signals, a process called
demultiplexing (or demuxing).

While we’re at it, we’ll also add support for encrypted communication, which
will secure our moon-earth communications so that anyone cannot eavesdrop our
messages.

Let’s add two new dependencies:

$ npm install --save libp2p-mplex@^0.8.5 libp2p-secio@^0.11.1

And we'll need to edit our bundle. Open src/libp2p_bundle.js and import the
new modules:

const Multiplex = require('libp2p-mplex') 
const SECIO = require('libp2p-secio')

Then change the DEFAULT_OPTS constant to look like this:

const DEFAULT_OPTS = {
  modules: {
    transport: [
      TCP
    ],
    connEncryption: [
      SECIO
    ],
    streamMuxer: [
      Multiplex
    ]
  }
}

That’s it! Now we can open multiple independent streams over our single TCP
connection, and our connection will be upgraded to a securely encrypted channel
using the secio module.

Let’s go interplanetary!

As we have now added all the necessary components for our communication, let’s
build our earth node.

Add the following code in src/earth.js

'use strict'
/* eslint-disable no-console */
const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
const async = require('async')
const chalk = require('chalk');
const emoji = require('node-emoji')
let moonPeerId
async.parallel([
    (callback) => {
        PeerId.createFromJSON(require('./ids/earthId'), (err, earthPeerId) => {
            if (err) {
                throw err
            }
            callback(null, earthPeerId)
        })
    },
    (callback) => {
        PeerId.createFromJSON(require('./ids/moonId'), (err, moonPeerId) => {
            if (err) {
                throw err
            }
            callback(null, moonPeerId)
        })
    }
], (err, ids) => {
    if (err) throw err
    const earthPeerInfo = new PeerInfo(ids[0])
    earthPeerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/0')
    const nodeDialer = new Node({ peerInfo: earthPeerInfo })
    const moonPeerInfo = new PeerInfo(ids[1])
    moonPeerId = ids[1]
    moonPeerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    nodeDialer.start((err) => {
        if (err) {
            throw err
        }
        console.log(emoji.get('large_blue_circle'), chalk.blue(' Earth Ready '),
                    emoji.get('headphones'), chalk.blue(' Listening on: '));
        nodeDialer.dialProtocol(moonPeerInfo, '/chat/1.0.0', (err, conn) => {
            if (err) {
                throw err
            }
            console.log('\n' + emoji.get('large_blue_circle'),
                        chalk.blue(' Earth dialed to Moon on protocol: /chat/1.0.0'));
            console.log(`${emoji.get('incoming_envelope')}
                         ${chalk.bold(`Type a message and press enter. See what happens...`)}`)
        })
    })
})

And similarly to the moon peer, we also have to create a JSON file containing
the peerId of our earth peer.

{
    "id": "Qma3GsJmB47xYuyahPZPSadh1avvxfyYQwk8R3UnFrQ6aP",
    "privKey": "CAASpwkwggSjAgEAAoIBAQCaNSDOjPz6T8HZsf7LDpxiQRiN2OjeyIHUS05p8QWOr3EFUCFsC31R4moihE5HN+FxNalUyyFZU//yjf1pdnlMJqrVByJSMa+y2y4x2FucpoCAO97Tx+iWzwlZ2UXEUXM1Y81mhPbeWXy+wP2xElTgIER0Tsn/thoA0SD2u9wJuVvM7dB7cBcHYmqV6JH+KWCedRTum6O1BssqP/4Lbm2+rkrbZ4+oVRoU2DRLoFhKqwqLtylrbuj4XOI3XykMXV5+uQXz1JzubNOB9lsc6K+eRC+w8hhhDuFMgzkZ4qomCnx3uhO67KaICd8yqqBa6PJ/+fBM5Xk4hjyR40bwcf41AgMBAAECggEAZnrCJ6IYiLyyRdr9SbKXCNDb4YByGYPEi/HT1aHgIJfFE1PSMjxcdytxfyjP4JJpVtPjiT9JFVU2ddoYu5qJN6tGwjVwgJEWg1UXmPaAw1T/drjS94kVsAs82qICtFmwp52Apg3dBZ0Qwq/8qE1XbG7lLyohIbfCBiL0tiPYMfkcsN9gnFT/kFCX0LVs2pa9fHCRMY9rqCc4/rWJa1w8sMuQ23y4lDaxKF9OZVvOHFQkbBDrkquWHE4r55fchCz/rJklkPJUNENuncBRu0/2X+p4IKFD1DnttXNwb8j4LPiSlLro1T0hiUr5gO2QmdYwXFF63Q3mjQy0+5I4eNbjjQKBgQDZvZy3gUKS/nQNkYfq9za80uLbIj/cWbO+ZZjXCsj0fNIcQFJcKMBoA7DjJvu2S/lf86/41YHkPdmrLAEQAkJ+5BBNOycjYK9minTEjIMMmZDTXXugZ62wnU6F46uLkgEChTqEP57Y6xwwV+JaEDFEsW5N1eE9lEVX9nGIr4phMwKBgQC1TazLuEt1WBx/iUT83ita7obXqoKNzwsS/MWfY2innzYZKDOqeSYZzLtt9uTtp4X4uLyPbYs0qFYhXLsUYMoGHNN8+NdjoyxCjQRJRBkMtaNR0lc5lVDWl3bTuJovjFCgAr9uqJrmI5OHcCIk/cDpdWb3nWaMihVlePmiTcTy9wKBgQCU0u7c1jKkudqks4XM6a+2HAYGdUBk4cLjLhnrUWnNAcuyl5wzdX8dGPi8KZb+IKuQE8WBNJ2VXVj7kBYh1QmSJVunDflQSvNYCOaKuOeRoxzD+y9Wkca74qkbBmPn/6FFEb7PSZTO+tPHjyodGNgz9XpJJRjQuBk1aDJtlF3m1QKBgE5SAr5ym65SZOU3UGUIOKRsfDW4Q/OsqDUImvpywCgBICaX9lHDShFFHwau7FA52ScL7vDquoMB4UtCOtLfyQYA9995w9oYCCurrVlVIJkb8jSLcADBHw3EmqF1kq3NqJqm9TmBfoDCh52vdCCUufxgKh33kfBOSlXuf7B8dgMbAoGAZ3r0/mBQX6S+s5+xCETMTSNv7TQzxgtURIpVs+ZVr2cMhWhiv+n0Omab9X9Z50se8cWl5lkvx8vn3D/XHHIPrMF6qk7RAXtvReb+PeitNvm0odqjFv0J2qki6fDs0HKwq4kojAXI1Md8Th0eobNjsy21fEEJT7uKMJdovI/SErI=",
    "pubKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaNSDOjPz6T8HZsf7LDpxiQRiN2OjeyIHUS05p8QWOr3EFUCFsC31R4moihE5HN+FxNalUyyFZU//yjf1pdnlMJqrVByJSMa+y2y4x2FucpoCAO97Tx+iWzwlZ2UXEUXM1Y81mhPbeWXy+wP2xElTgIER0Tsn/thoA0SD2u9wJuVvM7dB7cBcHYmqV6JH+KWCedRTum6O1BssqP/4Lbm2+rkrbZ4+oVRoU2DRLoFhKqwqLtylrbuj4XOI3XykMXV5+uQXz1JzubNOB9lsc6K+eRC+w8hhhDuFMgzkZ4qomCnx3uhO67KaICd8yqqBa6PJ/+fBM5Xk4hjyR40bwcf41AgMBAAE="
}

Here in earth.js, we used the same modules as we used in moon.js. The only
new module we use here is async, which is used to generate peerIds for moon
and earth peers, in parallel.

Then, as we did in moon.js we create the earthPeerInfo, add a
multiaddr,
/ip4/127.0.0.1/tcp/0 and finally, create our earth node nodeDialer.

Notice that we also, create moonPeerInfo and add the same
multiaddr, that we added
in moon.js. We do this as the earth node needs to know about the
moonPeerInfo in order to dial our moon peer. So, the moonPeerInfo acts
like a phone number here.

And finally, we start our earth peer and dial the moon peer using
nodeDialer.dialProtocol. While dialing to our moon peer, we need to specify a
protocol (/chat/1.0.0), using which we will talk to our moon peer. It's like
deciding a common language before we start talking to each other. Otherwise, it
would make no sense.

Now, as earth peer hash defined that it will use /chat/1.0.0 protocol to
communicate with the moon peer, let's add that to the moon.js as well.

To do that, we need to install a few more libs:

$ npm i pull-stream@^3.6.9 pull-pushable --save

Now, replace the moon.js with the following code:

'use strict'
/* eslint-disable no-console */
const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
const pull = require('pull-stream')
const Pushable = require('pull-pushable')
const p = Pushable()
const chalk = require('chalk');
const emoji = require('node-emoji')
PeerId.createFromJSON(require('./ids/moonId'), (err, peerId) => {
    if (err) {
        throw err
    }
    const peerInfo = new PeerInfo(peerId)
    peerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    const nodeListener = new Node({ peerInfo })
    nodeListener.start((err) => {
        if (err) {
            throw err
        }
        nodeListener.on('peer:connect', (peerInfo) => {
            console.log(emoji.get('moon'), chalk.blue(' Moon found Earth '),
                        emoji.get('large_blue_circle'),
                        chalk.blue(` on: ${peerInfo.id.toB58String()}`));
            console.log('\n' + emoji.get('moon'),
                        chalk.green(' Moon waiting for message from Earth ')
                        + emoji.get('large_blue_circle'))
        })
        nodeListener.handle('/chat/1.0.0', (protocol, conn) => {
            pull(
                p,
                conn
            )
            pull(
                conn,
                pull.map((data) => {
                    return data.toString('utf8').replace('\n', '')
                }),
                pull.drain(console.log)
            )
        })
        console.log(emoji.get('moon'), chalk.blue(' Moon ready '),
                    emoji.get('headphones'), chalk.blue(' Listening on: '));
        peerInfo.multiaddrs.forEach((ma) => {
            console.log(ma.toString() + '/p2p/' + peerId.toB58String())
        })
        console.log('\n' + emoji.get('moon'),
                    chalk.blue(' Moon trying to connect with Earth '),
                    emoji.get('large_blue_circle'));
    })
})

Now, as have defined /chat/1.0.0 protocol for moon peer too, let's try to
connect earth peer to our moon peer.

First, run node moon.js in one terminal. Then fire up another terminal and run
node earth.js.

If everything went right, then you would see something like below:

In case you are stuck somewhere, let us know here.

Let’s Talk to the Moon

Till now we have managed to connect moon peer and earth peer. Another
interesting thing that we can do here to allow these 2 peers to communicate with
each other.

We will use the command line to write and send our messages.

To do that we need to handle and send messages from the command prompt. We can
do that using process module available in NodeJS.

Replace moon.js with the following code:

'use strict'
/* eslint-disable no-console */

const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
const pull = require('pull-stream')
const Pushable = require('pull-pushable')
const p = Pushable()
const chalk = require('chalk');
const emoji = require('node-emoji')

PeerId.createFromJSON(require('./ids/moonId'), (err, peerId) => {
    if (err) {
        throw err
    }
    const peerInfo = new PeerInfo(peerId)
    peerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    const nodeListener = new Node({ peerInfo })

    nodeListener.start((err) => {
        if (err) {
            throw err
        }

        nodeListener.on('peer:connect', (peerInfo) => {
            console.log(emoji.get('moon'), 
                        chalk.blue(' Moon found Earth '),
                        emoji.get('large_blue_circle'),
                        chalk.blue(` on: ${peerInfo.id.toB58String()}`));
            console.log('\n' + emoji.get('moon'),
                        chalk.green(' Moon waiting for message from Earth ')
                        + emoji.get('large_blue_circle'))
        })

        nodeListener.handle('/chat/1.0.0', (protocol, conn) => {
            pull(
                p,
                conn
            )

            pull(
                conn,
                pull.map((data) => {
                    return data.toString('utf8').replace('\n', '')
                }),
                pull.drain(console.log)
            )

            process.stdin.setEncoding('utf8')
            process.openStdin().on('data', (chunk) => {
                var data = `${chalk.blue("Message received from Moon: ")}\n\n`
                + chunk.toString() + `\n${emoji.get('incoming_envelope')}
                ${chalk.blue("  Send message from Earth:")}`

                p.push(data)
            })
        })

        console.log(emoji.get('moon'), chalk.blue(' Moon ready '),
                    emoji.get('headphones'), chalk.blue(' Listening on: '));

        peerInfo.multiaddrs.forEach((ma) => {
            console.log(ma.toString() + '/p2p/' + peerId.toB58String())
        })

        console.log('\n' + emoji.get('moon'), chalk.blue(' Moon trying to connect with Earth '),
                    emoji.get('large_blue_circle'));
    })
})

Replace earth.js with the following code:

'use strict'
/* eslint-disable no-console */

const PeerId = require('peer-id')
const PeerInfo = require('peer-info')
const Node = require('./libp2p_bundle')
const pull = require('pull-stream')
const async = require('async')
const chalk = require('chalk');
const emoji = require('node-emoji')
const Pushable = require('pull-pushable')
const p = Pushable()
let moonPeerId

async.parallel([
    (callback) => {
        PeerId.createFromJSON(require('./ids/earthId'), (err, earthPeerId) => {
            if (err) {
                throw err
            }
            callback(null, earthPeerId)
        })
    },
    (callback) => {
        PeerId.createFromJSON(require('./ids/moonId'), (err, moonPeerId) => {
            if (err) {
                throw err
            }
            callback(null, moonPeerId)
        })
    }
], (err, ids) => {
    if (err) throw err
    const earthPeerInfo = new PeerInfo(ids[0])
    earthPeerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/0')
    const nodeDialer = new Node({ peerInfo: earthPeerInfo })

    const moonPeerInfo = new PeerInfo(ids[1])
    moonPeerId = ids[1]
    moonPeerInfo.multiaddrs.add('/ip4/127.0.0.1/tcp/10333')
    nodeDialer.start((err) => {
        if (err) {
            throw err
        }

        console.log(emoji.get('large_blue_circle'), chalk.blue(' Earth Ready '),
                    emoji.get('headphones'), chalk.blue(' Listening on: '));

        nodeDialer.dialProtocol(moonPeerInfo, '/chat/1.0.0', (err, conn) => {
            if (err) {
                throw err
            }
            console.log('\n' + emoji.get('large_blue_circle'),
                        chalk.blue(' Earth dialed to Moon on protocol: /chat/1.0.0'));
            console.log(`${emoji.get('incoming_envelope')}
                         ${chalk.bold(`Type a message and press enter. See what happens...`)}`)
            // Write operation. Data sent as a buffer
            pull(
                p,
                conn
            )
            // Sink, data converted from buffer to utf8 string
            pull(
                conn,
                pull.map((data) => {
                    return data.toString('utf8').replace('\n', '')
                }),
                pull.drain(console.log)
            )

            process.stdin.setEncoding('utf8')
            process.openStdin().on('data', (chunk) => {
                var data = chunk.toString()
                var data = `${chalk.blue("Message received from Earth: ")}\n\n`
                + chunk.toString() + `\n${emoji.get('incoming_envelope')}
                ${chalk.blue("  Send message from Moon:")}`

                p.push(data)
            })
        })
    })
})

That’s all. First, run node moon.js in one terminal. Then fire up
another terminal and run node earth.js.

Now, if everything in fine, then you will see the earth peer asking you to
type something and press enter. If you do that, you can see the same message
is received on the moon peer. Also, if you type something and press enter from
the moon peer, you can see the same message on the earth peer.

Congratulations🎉🎉 You are one of the very few people to communicate from earth
to moon!!

In case you are stuck somewhere, let us know here.

This article was first published on
SimpleAsWater.com.

Forem: vasa

Under the hood: AvionDB P2P Syncing

A peer-to-peer to-do list

Creating, listing & updating to-do(s)

Add To-do(s)

Get To-do(s)

Update To-do(s)

Syncing the to-do list across different peers

Finding Addresses

Common Protocol for Communication

Circuit Relay

How does circuit relay work?

What's up with this HOP and STOP?

A few caveats (and features)

A word on circuit relay addresses

Establish a connection

Sync data between the peers

AvionDB: A MongoDB-like Distributed Database

Enter AvionDB

Design Goals

How does it work?

Using AvionDB with NodeJS

Install

Usage

Using AvionDB with Browser

through Browserify

through webpack

from CDN

Usage

Demo Example of Firebase Authentication with AvionDB

Frequently Asked Questions

Are there implementations in other languages?

Where can I see your Roadmap?

What mongodb features does aviondb support?

Other Questions?

Contributing

Contact Us

How to Write Upgradable Smart Contracts (Smart Contract Versioning)

Why do we need to upgrade?

How do upgrades work under the hood?

OpenZeppelin Upgrades

OpenZeppelin Upgrades CLI

Project Setup

Writing & deploying contracts

Upgrading Contracts Programmatically

A few things to keep in mind: limitations & workarounds

Limitation 1: No Constructors

Workaround: Initializer

Limitation 2: Initial Values in Field Declarations

Workaround: Initializer

Limitation: Creating New Instances From Your Contract Code

Workaround: Inject a pre-deployed contract from CLI

Workaround: OpenZeppelin App Contract

Potentially Unsafe Operations

Modifying Your Contracts

References

Tutorial: Building a Twitter Bot using IPFS Cluster 🤖

Intro to IPFS Cluster

Setting up Your IPFS Peer

Setting up Your IPFS Cluster Peer

Making a few changes to IPFS Cluster Config

Starting Your IPFS Cluster Network

Creating a Twitter Developer Account

Building the Twitter Bot 🤖

Installing Golang

Writing the Twitter Bot Code 👩‍💻👨‍💻

Final Showdown

Hands-on IPLD Tutorial in Golang: PART 2

Quick recap from PART-1

What We Will Learn from PART-2? 🤔

Changing the Data Structures 🛠

Changing the Way We Track Entries 🧐

Taking User Input 👨‍💻👩‍💻

The Document-Storage Magic 🔮✨

A Change in Reading DAGs for Entries

Conclusion

About the Authors:

Ganesh Prasad Kumble

Vaibhav Saini

Hands-on IPLD Tutorial in Golang: PART 1

What's up with this `HOP` and `STOP`?