Forem: Eric Khun

Explaining "Proof of Work" Blockchains to Developers

Eric Khun — Thu, 08 Jul 2021 04:03:23 +0000

This post was originally posted in my personal blog.

What is a blockchain? Why are some people using it? Even as a developer, it took me a few years to understand what a blockchain really is,how it is working from inside out. I'll share here what I believe is a comprehensive explanation of what is a blockchain from a developer point of view.

What is a blockchain?

In simple words, it’s a slow database.

For Bitcoin, this database contains a ledger only. A ledger is basically records of how much “token” has been transferred between addresses. Everyone can create a “wallet”, and make a transaction on it. Those transactions will be visible by anyone in the world. Each time you want to “write” in the blockchain, it will cost a certain amount of “fees”.

“Dubious dev: So... why not create a huge Postgresql/MySQL/MongoDB database cluster, and make it open to everyone? 🤔”

What looks like a “blockchain”?

At a really high level, a blockchain is made of blocks containing some data. They are linked with each other with a reference (a hash) of its previous block:

It’s really similar to a linked list, but there are some fundamental differences between the two, notably the mechanism of how blocks are added and the data it contains.

How are “blocks” built?

The way new blocks get added is quite sophisticated and isn’t quite instant.

When you want to make a new transaction on the blockchain - let’s say someone wants to transfer 5BTC to someone else - this will create a new “transaction” but is not immediately added to the blockchain:

The transaction first gets added to the mempool
From this mempool, a miner will pick the transactions that he likes (the one that give him the most reward). The total size of all transaction transactions must be below 1MB.
The miner will run a “brute-force” algorithm to solve a puzzle. Once he find it, it will have the possibility to add the transactions he picked to the blockchain , but not always (I’ll come to that one later).

The Mempool is a sort of waiting room with a priority ticket for transactions that are willing to pay more. The ship contains only a few seats (1MB), so it’s up to the miner to decide if he let you in or not. But you might end up in a ship going nowhere.

You can check all the unconfirmed transactions waiting to be validated in the Bitcoin blockchain here. You can also read here a bit about the controversy to have a bigger block size .

Well, this mempool/miner/puzzle architecture looks quite complicated for a database. Why is it needed?

Protecting the blockchain with Consensus and Decentralization

A “classic” database will allow anyone who has its credential to add/remove/edit records at their will. Want to delete a transaction you’ve made? Add a higher amount in your own wallet? How would you know this database will always be managed the way it said it will? How do you make sure the one managing the system will never attempt to change the transactions or the data in it? This is where decentralization, consensus, and immutability comes into play.

The race to add a block to the chain proof and work

A blockchain “validates” and permanently stores the data after a consensus is reached. Consensus is an agreement about the state of the ledger. It agrees on things like account balances, the order of transactions, or even if transactions look legitimate. The most infamous consensus protocol is “Proof-of-Work” and used by Bitcoin and Ethereum.

Finding the “nonce”: Miners and Rewards

“Proof of Work” blockchains have an interesting concept where you have to solve a puzzle to be able to add a block to to chain. This puzzle is run by miners. It is a brute-force algorithm where the first miner who solved it gets a reward (and the privilege to add a block to the chain 😉). And this happens every time a new block needs to be added. Today, 1,000,000 miners are competing against each other in the hope to solve the puzzle.

Increasing difficulty

Bitcoin has a built-in algorithm to try to get a target of 10 minutes to mine a block. If it takes a lot shorter to mine, the difficulty increases, if it takes longer, difficulty decreases. It means the more “hashpower”, the more difficult is the hash function to find the nonce.

Since 2017, the difficulty has exponentially increased, requiring more powerful computation resource if you want to be lucky enough to mine a block. This is what lead to today’s heated discussion related to climate change and energy consumption:

How long would that take to mine a block with my computer?

A difficulty of 1 corresponds to 2^32 = 4,3 trillion hashes. Your computer will have to hash a random number (via bruteforce) 4.3T times in the hope to find the nonce with the lowest difficulty possible.

If we take today’s difficulty of about 20 trillions we will have to to go through 8.6 × 10^22 (it’s 86000000000000000000000 trillions 🤪) operation to try.

Now let’s use a standard processor a core i7 2600 with pretty good hashing power of 23.9 MHash per second, or 2.3e7 hashes/second (or 23,000,000 operation/second). It would take the computer 8.6 × 10^22 / 2.3e7 = 3.7e15 seconds to try all the possibilities, or about 118,567,000 years. In comparison, if the difficulty was at 1 (when Bitcoin was created) it only took 186 seconds to go through all the possibilities and find the nonce. I know, we would all be rich now right?

This difficulty increase is why miners buy more and more GPUs (CPUs aren't fast enough) to put in those gigantic “mining warehouses”, in the hope to find that lucky number.

Some new consensus starts to emerge to solve those issues. Ethereum is currently moving to a “Proof-of-Stake” consensus. The current Ethereum blockchain will "merge" (in 2022?) into the proof-of-stake system. Some (really) interesting variations of the Proof of Stake consensus are adopted in the blockchain space, particularly the Avalanche and Tendermint Byzantine Fault Tolerance consensus. Bitcoin however will keep the proof-of-work mechanism.

Securing the blockchain with decentralization and cryptography

“Dubious dev: What if I solve the puzzle and add a fake transaction 🤔?”

It's almost impossible to create new blocks that erase transactions or create fake ones. If you were lucky enough to find the nonce, this doesn’t mean it will be added to the blockchain:

You will broadcast the information to other other miners
Other miners will check carefully if the transactions if the block submit are actually legit transactions.
If 51% agree on the legitimacy of the block, consensus is reached, and the block is permanently added

The 2nd step is where the “crypto” in “cryptocurrency: plays its main role. Blockchains use cryptography functions in many ways, but the first use case is to make sure one’s doing a transaction is actually the “owner” of the transaction. It is also called signing a transaction. You can do that with a “secret key” (or commonly called private key). You can “sign” transactions with that private key to finalize an action. It’s like a real-world signature. In the cryptographers world, it is called “asymmetric cryptography” or “public-key cryptography” in the developer's world. With Bitcoin and Ethereum, miners will use the ECDSA (Elliptic Curve Digital Signature Algorithm) function to make that check.

More than value transaction

Some blockchains can store and execute some code, such as the Ethereum blockchain. The Ethereum “database” can store code and get the code executed. The code store is called a “smartcontract”. In Ethereum, the programming language is “Solidity”. For example:

receiveMoneyEvent(amount) {
    facilitatortFee = amount * 2/100
    splitMoneyAmount = (amount - facilitatortFee) / 2
    sendTo(charityAddressX, splitMoneyAmount)
    sendTo(charityAddressY, splitMoneyAmount)
    sentTo(facilitatortAddress, facilitatortFee)
}

In this example, each time the smart contract receives money, it will forward it to 2 charities. In between, the “facilitator” will take a 2% fee. Pretty simple isn’t it?

Once you publish this code to the blockchain ( you’ll have to pay each time you want to do that, it’s like an expensive git push command), anyone who wants to can call this contract. In this smartcontract, addresses that will send Ether (ETH) to this contract, those actions will be executed.

Smart Contract Immutability

The logic in those contracts never changes and will be on-chain forever. The contract you agree to interact will always do what it said it will. No one will ever be able to change the logic. The ones creating this contract won’t be able to change it, even if they published it. So there is no risk of 1 party changing terms, because the contract can’t be changed. It will be on the chain as long as the network is alive.

As a developer, you might think it’s a bad thing. What if there is a bug? Well you better write perfect code 😉. There are many ways to “hack” smart contracts. Some (really expensive) services , often owned by famous security companies, exist to review your smart contracts before you publish them.

Transparency (or Privacy?)

Privacy is one of the hottest topics in 2021. We don’t want to be tracked anymore, and try to hide from Internet giants like Facebook or Google. Bitcoin or Ethereum are however fully transparent. The ledger is public and anyone that has your wallet address can see the full history. It’s like a database with read privilege where you can query any balance or operations that happened for any wallet. For example see here how many Ethereum has Vitalik (Ethereum's dad), or how he received $375 millions worth of “AKITA” token, and then forwarded that "unwanted money) to an India Covid relief fund:

If you're looking for privacy, some blockchains such as zCash or Monero make that information hidden. (Read on to understand how). But they're still public since anyone can use them.

Some blockchains are private only. An interesting use case could be sharing some confidential data, between a group of specific persons. An idea could be applied for the CRS (Common Reporting Standard, for preventing tax avoidance). Governments could share information of bank accounts into that blockchain. With decentralization, no government will have the "privilege" to delete transactions in it. If a change needs to be made, the full history of changes will be written in the ledger.

Conclusion

A Blockchains is giant database that stores a ledger with all transactions of its users. It sometimes stores code that can be executed. The decentralization and consensus algorithm built into them make them almost impossible to be compromised by “bad actors”. With today’s blockchains aggregated valuation ($1.39Trillion!), it feels like the world is somehow betting that blockchains could be tomorrow's new way to exchange money and share information. Or is that valuation only blowing smoke 🤔?

This post was originally posted in my personal blog.

A beginner guide to setup an Ethereum full node

Eric Khun — Thu, 10 Oct 2019 19:01:31 +0000

This is a step by step guide to setup your own Ethereum node on the Ropsten network. At the end of this guide you'll be able to:

Run your own ethereum node synced with the Ropsten testnet network
Making sure to keep the node in sync with the blockchain on the network
Be able to communicate via the JSON-RPC API of your own node
Using the same method to run your node on the main network

Why do you want run your own node?

Once your dAPP and Smart Contract are working locally, you'll want to test it on a real network with (almost) real condition.
There is a service called Infura, allowing you to communicate easily with Ropsten via json-rpc without having to setup your own node. However, Infura doesn't allow you to listen to events that your smart contract could trigger. Events are useful when you want start to do some fancy thing like real time data replication on a second database or simply notify your users in your dApp. If you do not need events, you should definitely use Infura :)
An other reason would be to mine some new block and earn some money. But I won't cover that topic here

Prerequisite :

Your own server. I have a Digital Ocean one. A good droplet to use is the one with 2GB memory and 50GB of diskspace. I also pick Ubuntu 16.04 for the OS. If you're a student, you can have a nice $50 free credits for testing purpose.
At least 10GB availlable of space disk. The Ropsten chain takes about 9.0GB on my server disk (4rd February 2018) . The "real" ethereum blockchain requires about 50GB of diskspace.

Step1: Setup geth (go ethereum)

geth will allow you to sync the Ethereum blockchain, but also and run a rpc client

To set that up :

sudo apt-get install software-properties-common
sudo add-apt-repository -y ppa:ethereum/ethereum
sudo apt-get update
sudo apt-get install ethereum -y

I'm using geth version: 1.7.3-stable for this guide

Some people might prefer parity. This guide will focus on geth

Step2: Sync your node with the Ropsten network

The sync will take about 4 hours. I'm running the command in a screen console, just in case I lose the ssh connection or want to do something else in the server. You can also use tmux if you prefer.

Optional : Before syncing it , make sure to remove any previous data you might have locally. (don't run that if you're in a middle of a sync, or you'll need to start again) :

geth --testnet removedb

Then sync the full ropsten blockchain network:

geth --testnet --fast --bootnodes "enode://20c9ad97c081d63397d7b685a412227a40e23c8bdc6688c6f37e97cfbc22d2b4d1db1510d8f61e6a8866ad7f0e17c02b14182d37ea7c3c8b9c2683aeb6b733a1@52.169.14.227:30303,enode://6ce05930c72abc632c58e2e4324f7c7ea478cec0ed4fa2528982cf34483094e9cbc9216e7aa349691242576d552a2a56aaeae426c5303ded677ce455ba1acd9d@13.84.180.240:30303" console

Those flags are really important for the first run. They are useful for:

--testnet : you'll sync with the Ropsten network. Alternatively you can use other flags
--fast : you're enabling fast syncing
--bootnodes : Through these bootnodes a node can join the network and find other nodes. The one I've given are valid for the 3rd Feb 2018, and might change in the future. Check the ropsten official repo to get the updated bootnodes.
console: I attached the console to know the current state of the sync . Without that, you'll be a bit blind on the progress of the sync

The Ethereum blockchain will store the data in the default directory (/YOUR_HOME/.ethereum/testnet/geth/chaindata). You can also check the size of the blockchain with du -hs /YOUR_HOME/.ethereum/testnet/geth/chaindata

Note: geth will run indefinitely until you exit with ctrl-c. It needs to keep your node in sync with the network with all new data. So once you run the command, geth kept the --fast and --bootnodes xxxx in memory. So you can safely run geth (with always keeping the network you want to use , --testnet in that case)

(optional) Step2.1: restart the sync when `geth` error-out

It happened to me (several times) that geth errored-out during the sync. I didn't find out why, but simply run the following command, and the sync will continue from where it exited :

geth --testnet console

Note: I didn't specify the --bootnodes and --fast flags

If like me, geth kept erroring out, and you do not have time to monitor the sync, here a command you can run to run that command indefinitely if it errors-out:

until geth --testnet  console; do
  echo geth sync failed, retrying in 10 seconds...
  sleep 10
done

You can then come back few hours later without worrying if the sync went well :)
Also, having geth that keep exiting might be a sign that oyu do not have enough memory. I recommend at least 2GB.

(optional) Step2.2: Check the progress of the sync

Because you attached the console to the geth command with the console flag, you'll be able to know the progress of the sync by typing that following command inside the geth console :
> eth.syncing
It will return :

{
  currentBlock: 914011,
  highestBlock: 2579729,
  knownStates: 7383,
  pulledStates: 480,
  startingBlock: 914001
}

Note: highestBlock: 2579729 should match (even be greater) than the last block in Ropsten Etherscan website

Step3: Activate the JSON-RPC API (and keep your node in sync)

As a developer, sooner or later you'll want to start interacting with Geth and the Ethereum network via your own programs and not manually through the console. geth has built in support for a JSON-RPC based APIs (standard APIs and Geth specific APIs). These can be exposed via HTTP, WebSockets and IPC (unix sockets on unix based platforms).

geth --testnet --rpc --rpcapi "eth,net,web3" --rpccorsdomain '*' --rpcaddr 0.0.0.0 --rpcport 8080

--rpc Enable the HTTP-RPC server
--rpcaddr HTTP-RPC server listening interface
--rpcport HTTP-RPC server listening port
--rpcapi API's offered over the HTTP-RPC interface
--rpccorsdomain * for allowing everything access to it

When running that command, geth will keep your node in sync with the network. You can let that one run in a screen or tmux terminal to keep it alive. The best thing would be to let the process managed by supervisord or systemd

Step4: Query the JSON-RPC API

You can test if the api is working with curl:

curl -X POST --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":67}' http://0.0.0.0:8080

If all is fine, you should get a response that looks like :

{"jsonrpc":"2.0","id":67,"result":"3"}

(optional) Open the firewall

If you want your dApp or Metamask to communicate with your node, you'll need to open the port of your server. On Digital Ocean, it's quite easy via their console. With my example, I've opened the port 8080 in the Inbound settings.

Then you can query the api via :

curl -X POST --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":67}' http://X.X.X.X:8080

with replacing X.X.X.X the ip of your instance.

Also, please understand the security implications of opening up an HTTP/WS based transport before doing so! Hackers on the internet are actively trying to subvert Ethereum nodes with exposed APIs!

Run on the main Ethereum network

As many may have guess, you'll just need to remove the --testet flag to all commands I've shown you to be able to sync to the main Ethereum network! That's all!

Also, you can change the --testnet flag to --rinkeby to use the Rinkeby network. The main differences with Ropsten are detailed here.

What did I build with this?

With the Ethereum node, I've built Citymayor, a virtual game to trade cities around the world. This node was used to get all the events from our smart contract.

I hope this guide was useful! Please reach me if you have any questions! You can also see original post here