Originally I planned to release in December, but just before the holidays kicked into full gear, I found an issue with the HAF upgrade process during my testing. The HAF upgrade process enables a node to update its software without having to do a full replay (a full replay takes several days, whereas an upgrade completes in minutes or less typically).
While this problem wouldn’t cause any immediate issue for API node operators, as they’re going to have to do a full replay for this release anyways, it would have required API nodes to do more full replays again, if we found any further problems after the release.
So based on this, I decided to delay the official release of the HAF API node suite until after the holidays when all our devs were back and we had a fix for the upgrade issue (which we now have).
So we’ll be releasing it officially this week. It’s available now as 1.27.7rc16 for anyone who wants to get a jump on replaying it.
Also, here’s a quick summary on some of what the BlockTrades team has been working on since my last report. As usual, it’s not a complete list. Red items are links in gitlab to the actual work done.
Hived: blockchain node software
Mostly minor changes here recently.
Upcoming work will focus on an overhaul of the transaction signing system. Originally I had vague hopes we might be able to complete those changes in February, but realistically it’s likely to be at least some time in March/April time frame I think.
These changes will be included as part of the hardfork and they are also tightly related to the support for “Lite Accounts” using a HAF API (my plan is to offer a similar signing feature set across the two systems to keep things simple). So the Lite Account API will probably be rolled out on a similar time frame.
HAF: framework for creating new Hive APIs and apps
Most of the work done recently was performance optimizations made as a result of benchmark testing.
- Add index status tracking and new functions for HAF index management. This allows apps to add extra indexes to HAF tables and also tracks which apps needs those indexes. If an app registers the requested index before HAF finishes a replay, the index will be built non-concurrently along with the other HAF indexes. If an app requests it later, the index will be built concurrently to avoid disrupting the operation of other HAF apps.
- Add functions for apps to request/wait for full vacuums and allow for concurrent index creation during live sync During testing we found that some app tables that were heavily modified were growing in size a lot during an app’s replay. By periodically performing full vacuums on these tables, these tables get re-shrunk back to a reasonable size, and the apps replay much faster (around 2x faster in the case of reputation tracker, for example, if I recall correctly). Smaller tables also means the API calls that reference these tables are faster. This functionality is primarily useful for tables that receive a lot of UPDATE queries, as it clears out dead tuples.
- Analyze tables after index creation Instead of running 'analyze' in a separated transaction after all indexes are created during a HAF replay, now each table is analyzed in the same transaction that creates the indexes. This way apps are told 'HAF is ready' when HAF has finished the table analyzes and is therefore configured for maximal performance.
- Speedup processing associated with keyauth state provider
- Bug fixes and improves to HAF upgrade procedure This is the fix mentioned at the top of this post.
- Repairing threading bug involved with schemas
- Implement state change tracking for haf apps for performance reporting
Hivemind: social media API
A huge amount of work was done to hivemind as part of our final optimization work prior to release. Below represents only a small portion of this work:
- Full vacuum hive.posts after massive sync for better performance
- Speedup vacuuming after replay by only vacuuming hivemind tables and do it in concurrent threads
- Speedup json checking for API calls
- Speedup list_pop_communities
- Speed up queries by removing function call from filtering process
- Add author_id to posts parent_id_id_idx to allow get_account_posts_by_replies to do an index-only scan
- Reduce storage by removing list_comments API and associated table/indexes
- Set reasonable API return limits to reduce unnecessary server loading and prevent API amplification attacks
- Cleaned up and optimized many API calls (various merge requests)
HAfAH: account history API
- Automate publishing hafah images to remote registry when new images get tagged
- Grants to hafah_user needs to be on the end of installation to properly grant roles to backend functions, also fix rewriter rules
- Include case when signature is null in get_transaction
Balance tracker API: tracks token balance histories for accounts
- Push the postgrest-rewriter to docker registries for tagged releases
- Vacuum full account_balance_history table every 10m during massive sync
- Fix healthchecker issue
- Add application_name parameter to the postgres connection string to make it easier to see what app is doing what in pgAdmin and friends
Reputation tracker: API for fetching account reputation
- Do a vacuum full every time minutes during massive sync to keep size of account_reputations table reasonable
- Register HAF indexes using new index registry API
- Fix healthchecker issue
- Allow the docker health check for the block processor to report health
HAF Block Explorer
- vacuum full some of the balance tracker tables
- Register HAF indexes
- Rewrite and speedup block searching
- Replacing UNION with UNION ALL in the block processing function results in significantly improved query performance
- Remove cache from last synced block
WAX API library for Hive apps
I expect we’ll officially release the Typescript version of wax in the next couple of weeks. Below is a sample of recently completed work:
- Added support for account authority update operation
- Improvements to key leakage detection
- Working on Python version of wax: now it can create and sign transactions, but currently only offline version is supported.
- Continuing work on a generic UI component for the health-checker.
HAF API Node (Scripts for deploying and managing an API node)
- Tagged rc16 release candidate for all the software components for a HAF API node
- Increased cache time for some API calls based on production performance analysis
- Force nginx to parse and store the request body in a way that it can be logged better for performance analysis tools
- Allow hivemind's block processor 30s to shut down cleanly before killing it to prevent potential problems when interrupted during massive sync
- create_zfs_dataset creates an empty snapshot + snapshot_zfs_datasets checks if shared_memory.bin is ok
- Separate the hivemind installer so that it's not integrated into the block-processing container so that the postgrest server can start as soon as installer is finished
- Switch to PostgREST implementation of Hivemind API server
- Fix balance tracker so that it can be run standalone again
- Create a tool to simplify using specific versions of all apps (e.g. the current develop versions)
In related work, we also created a handy new tool for managing submodule dependencies across HAF and HAF apps: https://gitlab.syncad.com/hive/update_submodules
What's next?
We’ve essentially finished production testing for the rc16 HAF API node software, except for the final test of replacing it as our production software on api.hive.blog. We expect to complete that operation in the next few days (by Monday at the very latest).
We updated the release candidate testing server (https://api.syncad.com) to rc16 today, and I recommend all apps do final testing against this API node ASAP, prior to the switchover on https://api.hive.blog itself, to reduce the chance for a late discovery of compatibility issues that need to be adjusted for.
Once the software is officially released and running on api.hive.blog, I’ll make another post to provide some performance metrics for the new release and other information of interest.