Hardfork 26 has been live for more than a week, and the performance improvements and new features have been incredible across the board. It may seem a bit of a shock that we need to have another quick code upgrade two weeks later! Read on for details of why HF27 is already scheduled for an important fix.
If you missed the fact that the Hive codebase has just undergone Evolution for expansion of ease of onboarding, higher performance, lowered costs, and other great benefits, you can find the official explainer post here. After more than a week, the new Resource Credit delegation system and features like OBI (one block irreversibility), compression, and snapshots are drastically improving quality of life for projects and users in the ecosystem.
However, shortly after the new hardfork went live, witnesses discovered one aspect of the new codebase is not behaving exactly as it should: the block production schedule is randomly shuffling through backup witnesses improperly. Some backups are scheduled to produce a block each round (similar to a top 20 witness) for multiple rounds, while others are being skipped for much longer than they should be according to their rank. While this bug requires a quick hardfork to fix, it does not pose a threat to the blockchain, tokens and assets, and does not require a replay of infrastructure.
While there is no avenue for this shuffling issue to make it easier to attack the chain, it did impact reward incentives to backup witnesses, which could lead to a potential loss of backup witnesses in the long term. There's no doubt this has been a very frustrating time for backup witnesses who are waiting to have the shuffle land on them! It is important to note that all block production rewards were paid out at the correct rate and still at random to backup witnesses as scheduled: no additional rewards were paid, and none were missed. However, some backup witnesses have been lucky and have signed many more blocks than they normally would have, and some backup witnesses have not been scheduled at all when they normally would have been. It is very important to get this fixed as quickly as possible so that witness rank is properly impacting scheduling again, but not so fast that we disrupt exchanges, infrastructure providers, and users.
This fix has been in testing for a week now, and is ready to be deployed. If you check Hive's witness ranking lists, you will see some witnesses have already begun signalling Version 1.27.0. While there is very little by way of code that needed to be written to correct this shuffling issue, making sure the fix does not extend node downtime is paramount, as we are still having some infrastructure catching up after HF26.
♦️ What this means for witnesses:
You must upgrade to version 1.27.0
Hardfork 27 will not require a replay. You will be able to stop your node, upgrade, and restart. There will be almost no downtime required to apply the new version that includes the scheduling fix.
Find the 1.27.0 release tagged on the official repo here:
https://gitlab.syncad.com/hive/hive/-/tags/v1.27.0
HF27 will come into effect on or after Monday, October 24th at 12:00 UTC, when consensus is reached.
♦️ What this means for our exchanges and other types of infrastructure provider:
Hardfork 27 will not require any additional downtime. Much like witness nodes, exchange and other types of nodes will be able to stop, apply the new version, and restart. Our liaisons are working with exchanges and they have nearly completed their HF26 upgrades. Applying HF27 will not require an additional period of weeks of their nodes being kept offline.
♦️ What this means for end users:
You will not see any additional downtime, or much of anything at all! This hardfork does absolutely nothing above and beyond fixing the scheduling issue, as well as some small changes to snapshots, which are ways of recording the state of a node at a specific period in time for faster maintenance. There are no changes to any aspects of the core blockchain code apart from these "back end" fixes.
While we've had a spectacular run with smooth code deployment over the past few years, it is inevitable that there will be some bumps along the way.
Every hardfork is an opportunity to improve our testing regimen and to get more and more people contributing to code and bug hunting. This particular scheduling problem is overall harmless and does indeed still distribute block production rewards randomly, but definitely has a big impact on our backup witnesses and their block production rewards. It's fair to be concerned about chain security and to operate with that in mind, but equally important to make sure that Hive's decentralized distribution continues to all of the amazing block producers securing the chain from around the world.
Working as a team and a community in co-operation, competition, and collaboration simultaneously can be really difficult. How we handle getting better with each hardfork, bug, success and hurdle is part of what has made this blockchain and its users so resilient.