blocktrades update.png

We’ve recently added five new programmers that were previously working on another project to our Hive team: three frontend devs, one frontend tester, and one backend dev. We now have over 30 devs working on Hive projects.

Below are a few highlights of the Hive-related programming issues worked on by the BlockTrades team since my last report.

Hived: blockchain node software

Support for resuming artifact file generation. Also fixed flushing problems on long-running Hived nodes. This will be supplemented soon by: https://gitlab.syncad.com/hive/hive/-/merge_requests/974
Load snapshot clears shared memory file automatically
Unit testing improvements - developed new fixture much better emulating real Hived environment - most of Hive startup code has been reused:
https://gitlab.syncad.com/hive/hive/-/merge_requests/948
https://gitlab.syncad.com/hive/hive/-/merge_requests/963
https://gitlab.syncad.com/hive/hive/-/merge_requests/965
Fixed bugs in RC related condenser_api methods, to return legacy form of operation/transaction
Changes were made to the Hive requests/response verification schema library as required by clive. There were also changes in test-tools to allow clive to share code.
https://gitlab.syncad.com/hive/hive/-/merge_requests/959
https://gitlab.syncad.com/hive/hive/-/merge_requests/970
https://gitlab.syncad.com/hive/hive/-/merge_requests/972
https://gitlab.syncad.com/hive/test-tools/-/merge_requests/152
Reduced the size of proposal_pay_operation (unneeded data members removed). Resolves issue: https://gitlab.syncad.com/hive/hive/-/issues/498
Hive docker/run_hived_img.sh script improvements: https://gitlab.syncad.com/hive/hive/-/merge_requests/916
https://gitlab.syncad.com/hive/haf/-/merge_requests/300
https://gitlab.syncad.com/hive/hive/-/issues/498
https://gitlab.syncad.com/hive/hive/-/issues/548
https://gitlab.syncad.com/hive/hive/-/issues/547

Denser: modern replacement for Condenser

Although I am just announcing it now, the Denser project has actually been under way since the beginning of this year. The idea behind Denser is to replace the rather messy codebase of condenser (code used for hive.blog and several other hive-based front ends) with a cleaner, more maintainable codebase that uses the latest web technologies.

At this point, the Denser prototype has reached the stage where it is clear the project is very viable, so we’ve opened up the repo for general inspection by the community. Denser has an automated deployment system so I'm able to preview and test each new deployment of the site (but unfortunately that site isn't yet accessible to users outside of our local network).

Clive: new Hive wallet with a text-based user interface

Clive is a Hive wallet written in Python that runs on your own computer (it is not a web-based wallet where the code comes from a remote server) so it is inherently more secure than web-based wallets. Currently there are two such wallets available and supported in the Hive ecosystem: 1) a command-line interface wallet (aka the CLI wallet) written using C++ and 2) a graphical interface wallet called Vessel (a JavaScript-based wallet).

Clive is designed to be easier to use than the existing CLI wallet, but at the same time it doesn’t require a terminal that supports graphics like Vessel does. For most people, Clive should provide a more friendly interface for performing Hive operations in a high security environment compared to using the CLI wallet.

I saw the first demo of Clive on Friday and I’ll make a later post with some screenshots of it. Currently it only supports transfer operations, but the project is getting to the point where it should be easy to add support for many additional operation types soon.

Mostly work is focused on preparing a version for public demo purposes and bugfixing:

https://gitlab.syncad.com/hive/clive/-/merge_requests/84

Wax repo: glue code for C++, Python, and Javascript

Additional work related to Clive is also being done in the wax repo. Wax acts as a glue between the direct blockchain code (written in C++), clive-foundation (written in Python), and (soon) Javascript, providing a shared codebase for transaction serialization.

Beekeeper: tool for managing encryption keys and generating/signing Hive transactions

Along with the creation of Clive, we created a new C++ program that can be used to store encryption keys and sign transactions with those keys. The purpose of this new program is to separate the high-security aspects of encryption key management from other wallet operations. For example, Clive doesn’t directly store keys or sign transactions. Instead it communicates with a separate beekeeper process and requests these operations to be performed. For those familiar with the EOS ecosystem, beekeeper is based on the ideas embodied in keosd.

New WASM-based version of Beekeeper

The first version of Beekeeper was written in C++, but now we are creating a WASM (emscripten) toolchain based image to allow cross compiling beekeeper to a WASM target. The idea is to spawn beekeeper in a web browser to provide a common way to safely store keys, perform transaction serialization/deserialization and transaction signing.

The wasm version of beekeeper is in this branch currently: https://gitlab.syncad.com/hive/hive/-/tree/mt-wasm-beekeeper A merge request will be made soon (probably Monday).

WASM-based beekeeper is part of broader work related to creating a common authentication component for Denser, integrated chat, and all other apps where Hive authentication is needed.

The new WASM-based code will also eliminate the need to duplicate transaction serialization implementation in any Javascript library which wants to support Hive. The transaction generation code will soon be moved from beekeeper to the wax library to reduce the amount of code in beekeeper (so as to reduce the amount of security analysis required).

Hive Application Framework (HAF)

HAF is a SQL-based framework for creating highly scalable and robust 2nd layer apps that operate using data from the Hive blockchain.

New HAF features and bug fixes

provided common function to calculate HivePower from vests
fixes to query supervisor (due to bugs detected at using public haf instance): https://gitlab.syncad.com/hive/haf/-/merge_requests/291
https://gitlab.syncad.com/hive/haf/-/merge_requests/294
explicit error when custom tables have too long identifiers (and creating a shadow tables can fail silently)
unified way of creating forking and nonforking apps
Improved JSON conversion performance
Improved textual dump of hive.operation (to bytea literal) to speedup pg_dump/pg_restore processes
fixed bugs caused to HAF node crash while processing a fork
fixed database object ownership management for automatically created items (by HAF)
Fixes to account metadata state provider due to bugs detected while integrating it in the block explorer app

HAF query supervisor to prevent rogue queries consuming too many resources

One of the more important tasks we’ve been working on for HAF recently is the creation of a “query supervisor”. This is a set of code that is used to monitor the queries executed on a SQL server and limit how many resources can be used by that query before the query will be terminated.

The query supervisor is necessary to prevent potential denial-of-service attacks that might be launched against a HAF server, especially a HAF server that allows the execution of semi-arbitrary SQL code (e.g. the code of SQL code that would likely be used for a smart contract system running on a HAF server).

Publicly accessible read-only HAF server testing

Recently we setup an instance of a HAF server that is publicly accessible with the query supervisor enabled to rate limit loading on the database. By publicly accessible, I mean it is possible to make direct read-only SQL queries to this HAF database that are not mediated by any API interface.

Based on what we learned from that test, we’ve made further improvements to the query supervisor and plan to deploy a new version of the publicly accessible HAF server with the latest HAF version and the latest query supervisor in the coming week.

HAF bug fixes and test improvements

HAF-based block explorer

We’ve been working for a while on new HAF-based block explorer for Hive. There were two main reasons for creating another block explorer for Hive: 1) we wanted an open-source block explorer than could easily be deployed by any of the existing API node operators without adding much overhead to their servers as a means of further decentralizing this critical functionality and 2) we wanted a “heavy-duty” HAF app that could help us identify and fix potential weaknesses and programming difficulties that might arise when developing a complex HAF-based app.

Block explorer backend

Massive verification of account balances. This was done to verify the new features and recent fixes made to the HAF balance_tracker application which can provide historical data about various account balances.
Endpoints for saving balances
New endpoint for Block explorer UI

Block explorer UI

For the original prototype of the Block explorer UI, we only had one relatively new frontend programmer assigned. Now we’ve added several more experienced programmers to the project, and we’ve embarked on a major code rewrite to bring this project inline with the frontend coding guidelines we’re establishing for all our frontend projects in order to use a common set of web technologies.

We hope to complete the rewrite by the 3rd week of August. Here’s some of the work planned and the devs assigned to each piece:
https://gitlab.syncad.com/hive/block_explorer_ui/-/issues/7
https://gitlab.syncad.com/hive/block_explorer_ui/-/issues/8
https://gitlab.syncad.com/hive/block_explorer_ui/-/issues/9
https://gitlab.syncad.com/hive/block_explorer_ui/-/issues/10

Some upcoming tasks

Integration of keyauth state provider provided by HAF into HAF block explorer.
Continue work on Consensus State Providers (for more powerful HAF apps).
Continue work on HAF-based block explorer backend and GUI
Add support for more operations to Clive wallet.
Collect benchmarks for a hafah app operating in “irreversible block mode” and compare to a hafah app operating in “normal” mode (low priority).
Publish more documentation for various new tools (beekeeper, Clive, consensus state providers) and separate HAF documentation into smaller, more easily digestible chunks.
Create docker compose scripts to ease deployment of API node infrastructure.
Benchmark some of the recent performance improvements in HAF and HAF-based apps.
Deploy an updated version of HAF to our publicly-accessible HAF server.

5th update of 2023: including new apps Denser and WASM-based beekeeper, plus wax library