Harmony- Optimizing sharding technology to drive scalability & dec...

Introduction

After the invention of blockchain technology and cryptocurrency, things became easy for the people. For the first time in the history, people realized that absolute sovereignty of their asset can be possible, one does not have to be dependent upon a third party, further, a third party can not dictate a term upon a user, heavy transaction fees can be challenged. So this new technology thrives to disrupt the existing centralized system.

The blockchain technology opens up new possibilities and the various other types of assets which is not otherwise monetized until now can be monetized now(for example data trading with many sub-verticals) and can be value transferred with the essence of blockchain technology. There is a wide range of potential applications of this technology, but unfortunately, the applicability is not encouraging.

It does not matter how wonderful the features of a blockchain is, unless it is highly scalable, it won't be able to widen its applicability.

Many efforts have been put since 2009 and many types of solution have come. But by large the solution is by the twist of one fundamental parameter to achieve another goal. Scalability is being achieved by slightly twisting the decentralization and advocating for it to justify its applicability.

The other point of view is that if it is scalable but not decentralized properly, then why to choose blockchain technology at all. Don't we have had the highly scalable centralized platform?

So definitely the choice is not between scalability and decentralization. The real choice for the next generation is the choice of both- decentralization & scalability. Therefore instead of adjusting one parameter to achieve the other, it is better to focus on the core structure of blockchain technology and bring innovations at every layer of it so that blockchain can be fundamentally scalable and decentralized both. And then no one can say 'scalability is a generic bottleneck of blockchain technology'.

Harmony

Harmony is a permissionless public blockchain which is provably secure, energy-efficient, highly scalable and sufficiently decentralized. It optimizes the blockchain protocol by innovating the sharding technology further and deep shard the chain at the consensus layer.

ds04 (2).png

Harmony has a secure staking mechanism which evenly distributes the stake and the probability of an attack is further eliminated by resharding with decentralized randomness. The peer-to-peer networking within the network is optimized with cross-shard routing and it results in faster block propagation.

Main features of Harmony

Secure Sharding

Sharding is the mechanism in which the network is divided into a number of smaller groups and these groups are known as shards. These shards process transactions in parallel and increase the transaction throughput of the overall network.

Since 2017, Sharding as a scaling solution has been largely recognized as a viable 1st layer scaling solution. We have Zilliqa, Quarkchain which have adopted sharding technology in their structure. But their sharding is only at the transaction layer. So the nodes of the shards process a subset of the transactions of the entire network and their state is the same as the entire network.

Harmony also uses sharding in its structure, but it optimizes the sharding technology not just at the transaction layer but at every layer such as protocol layer, network layer, transaction layer. That is why it is known as state sharding or deep sharding. So the transactions and the state of all the shards will differ from each other. It can be tricky, can be complex and can have security challenges because the sharding is being done at every layer. But Harmony's security paradigm is robust enough to beat any malicious element. It allocates the nodes to different shards randomly so that for an attacker it is difficult to enter its structure.

Security of sharding structure is achieved in the following ways-

Using a predetermined time interval known as epoch(which is roughly 1 day) and changing the nodes of the shards in every epoch.
The change of nodes of a shard after an epoch ends and its re-allocation to other shards is done by randomness generation using DRG protocol, VDF(Verifiable Delay Function) delays the reveal of the random number to prevent the last-revealer attack.
It uses Beacon chain and Shard chain. For an attacker to enter into the structure, it has to corrupt both Beacon chain & Shard chain, which is not easy.
New nodes can join the network at the beginning of each epoch and using cuckoo-rule they are allocated to a particular shard.
Harmony shards by voting shares not by validators; Adaptive threshold PoS further adjusts the price of voting share, so it is almost impossible for the attacker to gain access to the network.

Highly scalable BFT consensus

The existing and older blockchains mostly use PoW consensus which is neither fast nor linear. Take, for example, BTC and ETH still use the same old PoW consensus, as a result, the scaling capacity is very low. In Harmony network PoW will not fit well; the reason being it is a sharding based network and choosing PoW consensus will expose the network to security challenges like 1% shard attack.

So Harmony does away with PoW and use PoS for node election and allocation of nodes to subsequent shards. But the consensus in Harmony is run through BFT algorithm. That is Byzantine Fault Tolerance, in which the basic concept is that the network can still run and reach consensus even though there are malicious validators. But the network has to keep such malicious validators below the threshold. Harmony's security assumption is to keep such malicious validators below 1/4th.

However, Harmony has made modifications to the existing PBFT mechanism to make it a fast BFT and that is why it has named its consensus as FBFT. That is Fast Byzantine Fault Tolerance.

In the PBFT, the consensus process is as follows-

The leader broadcast a message to all the validators.
The validators rebroadcast that message to all other nodes, for the purpose of counting votes.
The phase completes when there is at least {2*(number of malicious validators)+1} consistent votes.

Please note that there are two phases prepare & commit phase and the above consensus process is carried out in both the phases.

Now the network cost from running this consensus is that if there are n number of nodes, then the leader will send "n" messages to the validators and the validators will again send the "n" number of messages to the other validators nodes.

So the net overload on the network is- O(n * n).

Now consider this for Harmony, to bear in mind that Harmony is a sharded network, so with a network complexity of O(n * n), is it going to work for Harmony? Does a network complexity of O(n * n) support sharded network with hundreds or thousands of nodes in each shard?

The answer is No & due to this Harmony has made certain improvements over PBFT.

The improvement is that- For counting of votes, the nodes do not require to resend the message to other validators. The counting of votes in Harmony will be done by the leader through multi-signature signing.

So with this improvement, the overload in the network is now simple O(n) and this can very well work for a sharded network like Harmony, as the equation becomes linear.

Moreover, when it comes multi-signature signing, Harmony selects BLS signature over Schnorr signature. Schnorr signatures require two round trips for a single multi-signature signing, on the other hand, BLS will just require one round trip for the same. So using BLS over Schnorr signature further makes the process linear for Harmony.

So the summary is that Harmony does not use any chain based consensus. The consensus in Harmony is reached by running FBFT algorithm among all the validators and the counting of votes is done by multi-signature signing using BLS signature.

In Harmony using FBFT & BLS signature, the consensus is reached as follows-

The leader broadcasts the block header to all the validators.
The validators sign the block header and send it back to the leader.
The leader broadcasts the aggregated multi-signature with a bitmap when there is at least {2*(number of malicious validators)+1} valid signature. That completes the prepare phase.

As there are two phases prepare & commit, the same process of prepare phase is repeated in commit phase and finally the block is committed with all the multi-signatures with bitmap attached.

This FBFT with BLS signature scales linearly with the number of nodes and in a network like Harmony, it is ideal to its sharding structure.

DRG, VRF, VDF for node allocations to shards

A random number is generated using DRG protocol and that random number is used for allocation of nodes to the different shards before the start of an epoch. The randomness is defined with certain properties to make it secure against any attack.

The random number is-

Unpredictable.
Not biaseable by any participant.
Verifiable.
Scalable.

VRF(Verifiable Random function) is computed to create a random number using cryptography. VDF then delays the randomness revelation.

The DRG protocol used to generate randomness and the final randomness is committed in the following ways-

The leader sends the message with the hash of the last block to all the validators.
VRF is computed to generate a random number & a proof for all the validators. So if there are "n" number of validators then the random data set will be n(random number, proof). This data set is sent to the leader.
The leader from the data set gets the final randomness.
To achieve consensus on the final randomness BFT is run among the validators. After that, it is committed in the block.
Now the leader computes actual randomness which is delayed by VDF.
Once the actual randomness is computed, the leader again runs BFT to reach consensus on the validity of final randomness and then it is committed in the blockchain.

How good is DRG protocol in randomness generation against any malicious behaviour?

Ans- VRF will not let malicious behaviour to bias or predict any randomness. A corrupt leader can not manipulate the actual randomness. A timeout mechanism is also in place, which will switch the leader and restart the protocol if the final randomness is not committed.

Adaptive Threshold PoS

The adaptive threshold is an important security requirement in the sharded structure of Harmony. While PoS is used for the election of validators, the assigning of voting share is not based on validators rather it is based on shards. The price of the voting share is also adaptively adjusted to keep the probability of attack very low.

The price of the voting share is calculated as-

Stake based sharding

The validators are elected based on staking. The number of tokens determines the voting share, That means the voting share is proportional to staked tokens.
The voting shares are then randomly assigned to shards.
The stakers who become validators by staking, goes to that shard where their voting shares are assigned.

Resharding

Resharding is done in Harmony network using Cuckoo rule. Any new node which wants to join can join the network at the start of the epoch by staking. The new nodes get new voting shares. The allocation of these voting shares is done in this way-

The voting shares are assigned to the shards which have more than median of the total voting shares.
A constant number of voting shares from all the shards are then re-distributed randomly to the rest fo shards which have less than the median of the total voting shares.

Beacon chain & Shard chain

The various shards that exist in the sharded network of Harmony are known as shard chains which have their own state and they process their own set of transaction.

Beacon chain also similar to shard chain but it has some additional responsibilities-

Generate randomness
Accept stakes from those who want to become validators.

Faster Cross-Shard Communication

It is relatively more important for better communication; especially faster & simpler communication for sharded network like Harmony.

In any normal public blockchain, the communication between the node is through the conventional gossip protocol. In which one node sends the same message to other nodes. But in a sharded network like Harmony, this type of gossip protocol will not be effective and the communication can not be made faster between the shards. So Harmony uses a different protocol which is known as Kademlia protocol.

Let us analyze the difference between the two-

In Gossip protocol. A node will send the same message to other nodes. Take, for example, there are "n" number of nodes, the network overload for message broadcast will be- n * O(message size)

Harmony uses erasure code and encode the message and sends the chunks of the message to the other nodes. As a result, the network overload for message broadcast is O(size of erasure code+message size).

So the network overload for message broadcasting is lower in Harmony as compared to gossip protocol message broadcast.

But Kademlia protocol has another distinct advantage.

Suppose the network overload in gossip protocol is O(n), then in Kademlia protocol, it will be O(log(n)).

The advantage of Kademlia protocol is that it has a "log of node distances" which is used for routing the message. But the important factor is that it travels and locate logarithmically and it eventually finds the shortest possible route for the message broadcast. Then the message is broadcasted only to a small group of nodes which fall on the way of the shortest path to the destination. So Kademlia definitely makes the cross-shard communication much faster than normal gossip protocol.

Tokenomics

ONE token is the native token of Harmony network.

The utilities of ONE token are-

It is used to stake in order to become a validator and earn block rewards.
It is used to pay the fees on the network such as transaction fee, gas fee, storage fee, etc.
It is used for voting in on-chain governance.

Potential Applications of Harmony Network

Harmony network has been designed to favor a market place where thousands of dppps can work without any congestion. It can handle millions or billions of people. So the large economies can greatly benefit from this infrastructure. Gaming platform can find it as a better alternative. The blockchain applicability is still very limited and its potential is mostly underutilized. Harmony aims to make a radically fair economy on the top of blockchain technology.

The data market is not an organized sector. Whatever presence it has, has limited sub-verticals. So the next generation is likely to witness data market as an organized sector. So the users using zk proof can value trade their data and most precisely it will have many verticals. Harmony as a blockchain infrastructure will strengthen this use case and will definitely give its due and provably establish the data market as an organized sector.

In a digitized world, when there will be an organized data market empowered with blockchain technology, people can trade their data in a ubiquitous way with the power of AI and blockchain technology and provably it can also ensure a general basic income for the common people.

Pros & Cons

Pros

Less network overload.
Faster communication between the shards
Linear and fast consensus using FBFT & BLS signature.
Easy synchronization for the new node who join the network.
Lower the barrier to entry as the total burden of the state of network is not needed for a node to participate.
Optimization at every level- so scalability greatly improved- testnet result is 118000 tps
Excels in all the department such as scalability, security & decentralization.
It is an open platform and foundation for all types of economies to grow-gaming, financial services, data sharing, decentralized economies.
Optimized sharding- deep sharding.
Low cost transaction
Energy efficient.

Cons

Complex sharding network.

Comparison

Review & rating

I have come across many types of blockchains and many types of scaling mechanism but for the first time, I can see a blockchain with state sharding technology. For the first time, I can also realize that state sharding is also possible. That is definitely a milestone. But the state sharding also comes with the challenges. And the Harmony team should be complemented more because of how they have designed the sharding structure and tackle the challenges.

I want to give a rating of 4.5 star out of 5 stars.

Conclusion

It is amazing to see a state sharding blockchain in 2019(the first-ever). While sharding in itself is a technology to improve scalability, Harmony team optimized the sharding to make it fully scalable. Since 2009 I have been hearing that scalability is a generic bottleneck, but at least now with Harmony network I can tell that scalability was never a generic bottleneck, rather blockchain was not explored to its potential and Harmony has brought that innovation now, the scalability was hidden there and Harmony has explored the truly scalable solution without twisting its decentralization parameter. But in a true sense, Harmony really wins it with its design of sharding structure and tackling the security challenges. I can see the potential of Harmony as a ubiquitous Blockchain infrastructure for Billions.

Harmony Team

More Information & Resources

Image Courtesy

Harmony.One

Harmony- Optimizing sharding technology to drive scalability & decentralization