get_transaction_hex and HIVE client libraries
There are multiple libraries for interacting with the HIVE blockchain, and these libraries seem to all share one property that from a distance seems rather silly, untill you look at the actual C++ source code for the HIVE blockchain and realize that with new hard forks important things might change.
When you are going to sign an operation on the HIVE blockchain, for example a simple vote on this post, a little bit of JSON is sent to one of the HIVE nodes, and added to this JSON there is a signature string.
Sounds simple enough were it not that the actual JSON itself isn't what is actually signed. Instead a packed variant of this JSON is to be signed, and how is the original JSON transaction converted into it's packed variant? It is commonly done server side on one of the public nodes.
There is an API call get_transaction_hex in the HIVE database API that does just that.
That API call takes the transaction JSON, sends it to an node, and gets back a hexadecimal representation of the packed version of the JSON.
why look at get_transaction_hex ?
Those of you that have been on HIVE a little bit longer may remember my old asyncsteem python library back in the STEEM days. It was a Python 2 library built on top of the Twisted networking library, a bit archaic for todays standard, but it was fully async, as opposed to lighthive by @emrebeyler, that is a pretty decent and pythonic HIVE library that while using async operations internaly so far doesn't seem to expose an async API, and beem by @holger80 that is not ll that bad but not as pythonic as lighthive, and it seems no effort towards an async future.
After recent problems with my little HIVE-archeology bot, that currently uses lighthive, and with coinZdense project in mind, I started to think that maybe I should start at a little HIVE python library myself, because there will be overlap between what a new async python lib for HIVE would need to do and what a more generic Web 3.0 coinZdense signing library would do.
Cloning the HIVE source code
The HIVE source code on github doesn't seem to be the right repo, at least not the main branch, instead the gitlab server on syncad seems to be the place to be.
So the first thing we need to do is fetch the hive source code:
git clone https://gitlab.syncad.com/hive/hive.git
The source file of interest
The source file that is of interest to us most at this point is the file hive_operations.hpp. This header file defines the operations that the HIVE API has, and then some (there are a few deprecated operations still there).
We arent't going to look at any other files, but this file has include files that have include files, so there will still end up being quite a bit of files that clang is going to use in order to do what we need it to do.
Setting the source code to hardfork 26
The next step is setting the codebase to hardfork 26. We do this by creating a sumlink:
ln -s ../../../hardfork.d/1_26.hf ./hive/libraries/protocol/include/hive/protocol/hardfork.hpp
dependencies
The code we will look at doean't have many dependencies, at least no non standard ones. On my ubuntu system I only needed to install libssl-dev in order to run clang on our header file.
sudo apt-get instll libssl-dev
Creating an AST JSON file from our C++ code.
Now we get to the fun part. C++ isn't really easy to parse for script code, but JSON is. The clang compiler though has an interesting feature that lets you compile C++ code into an Abstract Syntax Tree or AST. This isn't always a workable result because ASTs can get rather big, and written to JSOn they get even bigger.
Don't get startled by the length of the following commandline, we are just telling clang where to look for header files.
clang -Xclang -ast-dump=json hive/libraries/protocol/include/hive/protocol/hive_operations.hpp -I hive/libraries/protocol/include -I hive/libraries/fc/include --std=c++17 > mega-ast.json
We end up with a rather big JSON file. A little chunk of our file:
The total file is 31,700,210 lines long, 1.6 GB. Small enhough to handle in memory with scripts, but you can imagine that there are many scenarios where the whole AST becomes too big to handle.
What are we searching for?
Lets look at the start of our C++ file:
We see a number of structs that are derived from a baseclass and that live in a namespace (protocol) within another namespace (hive).
In the AST, a namespace is defined as NamespaceDecl, while structs are defines as CXXRecordDecl and individual member fields as FieldDecl.
Some python
So let's have some fun with this. A little bit of python:
#!/usr/bin/python3
import json
def process_opp_struct(obj):
rval = {"name": obj["name"], "fields": []}
for field in obj["inner"]:
if field["kind"] == "FieldDecl":
rval["fields"].append({"name": field["name"], "type": field["type"]["qualType"]})
return rval
def find_struct(lst):
for obj in lst:
if obj["kind"] =="CXXRecordDecl" and "bases" in obj:
match = False
for base in obj["bases"]:
if "type" in base and "qualType" in base["type"] and base["type"]["qualType"] == \
"hive::protocol::base_operation":
match=True
if match:
rval = process_opp_struct(obj)
if bool(rval["fields"]):
yield rval
def find_protocol(lst):
for obj in lst:
if obj["kind"] == "NamespaceDecl" and obj["name"] == "protocol":
for rval in find_struct(obj["inner"]):
yield rval
with open("mega-ast.json") as bigast:
data = json.load(bigast)
result = []
for obj in data["inner"]:
if obj["kind"] == "NamespaceDecl" and obj["name"] == "hive":
for res in find_protocol(obj["inner"]):
result.append(res)
print(json.dumps(result, indent=4))
Down below we itterate the top level namespace looking for anyghing in the hive namespace.
The find_protocol function looks for the protocol namespace. Then find_struct looks for structs that derive from the base_operation.
Finaly the process_opp_struct function looks if the struct has fields and what their type is.
A relatively simple script that we write once. If we were going to extract this data for a Python or Elixir or TypeScript or Clojure or whatever lib, it would be cumbersome but more importantly, we will need to put in a lot of effort when a new hardfork came out in order to stay up to date. Now we have a script we can just run again against a new hardfork, and we can use the resulting json as part of our implementation.
And it's not just about the packing to avoid calling get_transaction_hex API. Python allows us to write flexible API's that are transparant on top of dynamic definitions, for what we could use the JSON as input. The lighthive library does this already for other purposes. We will see this in a future post about the subject.
The result
I'm not going to further in this post, for once because I've not yet completed my packing efforts. But we are going to have a look at the results.
So this is the JSON rendering of the relevant parts ofh our C++ header file.
It is still a pretty big file, but not 32 million lines or 1.6GB of data. The result right now is 1060 lines or 28 kb of data.
[
{
"name": "account_create_operation",
"fields": [
{
"name": "fee",
"type": "hive::protocol::asset"
},
{
"name": "creator",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_account_name",
"type": "hive::protocol::account_name_type"
},
{
"name": "owner",
"type": "hive::protocol::authority"
},
{
"name": "active",
"type": "hive::protocol::authority"
},
{
"name": "posting",
"type": "hive::protocol::authority"
},
{
"name": "memo_key",
"type": "hive::protocol::public_key_type"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
}
]
},
{
"name": "account_create_with_delegation_operation",
"fields": [
{
"name": "fee",
"type": "hive::protocol::asset"
},
{
"name": "delegation",
"type": "hive::protocol::asset"
},
{
"name": "creator",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_account_name",
"type": "hive::protocol::account_name_type"
},
{
"name": "owner",
"type": "hive::protocol::authority"
},
{
"name": "active",
"type": "hive::protocol::authority"
},
{
"name": "posting",
"type": "hive::protocol::authority"
},
{
"name": "memo_key",
"type": "hive::protocol::public_key_type"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "account_update_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "owner",
"type": "optional<hive::protocol::authority>"
},
{
"name": "active",
"type": "optional<hive::protocol::authority>"
},
{
"name": "posting",
"type": "optional<hive::protocol::authority>"
},
{
"name": "memo_key",
"type": "hive::protocol::public_key_type"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
}
]
},
{
"name": "account_update2_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "owner",
"type": "optional<hive::protocol::authority>"
},
{
"name": "active",
"type": "optional<hive::protocol::authority>"
},
{
"name": "posting",
"type": "optional<hive::protocol::authority>"
},
{
"name": "memo_key",
"type": "optional<hive::protocol::public_key_type>"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
},
{
"name": "posting_json_metadata",
"type": "hive::protocol::json_string"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "comment_operation",
"fields": [
{
"name": "parent_author",
"type": "hive::protocol::account_name_type"
},
{
"name": "parent_permlink",
"type": "std::string"
},
{
"name": "author",
"type": "hive::protocol::account_name_type"
},
{
"name": "permlink",
"type": "std::string"
},
{
"name": "title",
"type": "std::string"
},
{
"name": "body",
"type": "std::string"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
}
]
},
{
"name": "comment_options_operation",
"fields": [
{
"name": "author",
"type": "hive::protocol::account_name_type"
},
{
"name": "permlink",
"type": "std::string"
},
{
"name": "max_accepted_payout",
"type": "hive::protocol::asset"
},
{
"name": "percent_hbd",
"type": "uint16_t"
},
{
"name": "allow_votes",
"type": "bool"
},
{
"name": "allow_curation_rewards",
"type": "bool"
},
{
"name": "extensions",
"type": "hive::protocol::comment_options_extensions_type"
}
]
},
{
"name": "claim_account_operation",
"fields": [
{
"name": "creator",
"type": "hive::protocol::account_name_type"
},
{
"name": "fee",
"type": "hive::protocol::asset"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "create_claimed_account_operation",
"fields": [
{
"name": "creator",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_account_name",
"type": "hive::protocol::account_name_type"
},
{
"name": "owner",
"type": "hive::protocol::authority"
},
{
"name": "active",
"type": "hive::protocol::authority"
},
{
"name": "posting",
"type": "hive::protocol::authority"
},
{
"name": "memo_key",
"type": "hive::protocol::public_key_type"
},
{
"name": "json_metadata",
"type": "hive::protocol::json_string"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "delete_comment_operation",
"fields": [
{
"name": "author",
"type": "hive::protocol::account_name_type"
},
{
"name": "permlink",
"type": "std::string"
}
]
},
{
"name": "vote_operation",
"fields": [
{
"name": "voter",
"type": "hive::protocol::account_name_type"
},
{
"name": "author",
"type": "hive::protocol::account_name_type"
},
{
"name": "permlink",
"type": "std::string"
},
{
"name": "weight",
"type": "int16_t"
}
]
},
{
"name": "transfer_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "amount",
"type": "hive::protocol::asset"
},
{
"name": "memo",
"type": "std::string"
}
]
},
{
"name": "escrow_transfer_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "agent",
"type": "hive::protocol::account_name_type"
},
{
"name": "escrow_id",
"type": "uint32_t"
},
{
"name": "hbd_amount",
"type": "hive::protocol::asset"
},
{
"name": "hive_amount",
"type": "hive::protocol::asset"
},
{
"name": "fee",
"type": "hive::protocol::asset"
},
{
"name": "ratification_deadline",
"type": "fc::time_point_sec"
},
{
"name": "escrow_expiration",
"type": "fc::time_point_sec"
},
{
"name": "json_meta",
"type": "hive::protocol::json_string"
}
]
},
{
"name": "escrow_approve_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "agent",
"type": "hive::protocol::account_name_type"
},
{
"name": "who",
"type": "hive::protocol::account_name_type"
},
{
"name": "escrow_id",
"type": "uint32_t"
},
{
"name": "approve",
"type": "bool"
}
]
},
{
"name": "escrow_dispute_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "agent",
"type": "hive::protocol::account_name_type"
},
{
"name": "who",
"type": "hive::protocol::account_name_type"
},
{
"name": "escrow_id",
"type": "uint32_t"
}
]
},
{
"name": "escrow_release_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "agent",
"type": "hive::protocol::account_name_type"
},
{
"name": "who",
"type": "hive::protocol::account_name_type"
},
{
"name": "receiver",
"type": "hive::protocol::account_name_type"
},
{
"name": "escrow_id",
"type": "uint32_t"
},
{
"name": "hbd_amount",
"type": "hive::protocol::asset"
},
{
"name": "hive_amount",
"type": "hive::protocol::asset"
}
]
},
{
"name": "transfer_to_vesting_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "amount",
"type": "hive::protocol::asset"
}
]
},
{
"name": "withdraw_vesting_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "vesting_shares",
"type": "hive::protocol::asset"
}
]
},
{
"name": "set_withdraw_vesting_route_operation",
"fields": [
{
"name": "from_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "to_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "percent",
"type": "uint16_t"
},
{
"name": "auto_vest",
"type": "bool"
}
]
},
{
"name": "witness_update_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "url",
"type": "std::string"
},
{
"name": "block_signing_key",
"type": "hive::protocol::public_key_type"
},
{
"name": "props",
"type": "hive::protocol::legacy_chain_properties"
},
{
"name": "fee",
"type": "hive::protocol::asset"
}
]
},
{
"name": "witness_set_properties_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "props",
"type": "flat_map<std::string, vector>"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "account_witness_vote_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "witness",
"type": "hive::protocol::account_name_type"
},
{
"name": "approve",
"type": "bool"
}
]
},
{
"name": "account_witness_proxy_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "proxy",
"type": "hive::protocol::account_name_type"
}
]
},
{
"name": "custom_operation",
"fields": [
{
"name": "required_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "id",
"type": "uint16_t"
},
{
"name": "data",
"type": "vector"
}
]
},
{
"name": "custom_json_operation",
"fields": [
{
"name": "required_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "required_posting_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "id",
"type": "hive::protocol::custom_id_type"
},
{
"name": "json",
"type": "hive::protocol::json_string"
}
]
},
{
"name": "custom_binary_operation",
"fields": [
{
"name": "required_owner_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "required_active_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "required_posting_auths",
"type": "flat_set<hive::protocol::account_name_type>"
},
{
"name": "required_auths",
"type": "vector<hive::protocol::authority>"
},
{
"name": "id",
"type": "hive::protocol::custom_id_type"
},
{
"name": "data",
"type": "vector"
}
]
},
{
"name": "feed_publish_operation",
"fields": [
{
"name": "publisher",
"type": "hive::protocol::account_name_type"
},
{
"name": "exchange_rate",
"type": "hive::protocol::price"
}
]
},
{
"name": "convert_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "requestid",
"type": "uint32_t"
},
{
"name": "amount",
"type": "hive::protocol::asset"
}
]
},
{
"name": "collateralized_convert_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "requestid",
"type": "uint32_t"
},
{
"name": "amount",
"type": "hive::protocol::asset"
}
]
},
{
"name": "limit_order_create_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "orderid",
"type": "uint32_t"
},
{
"name": "amount_to_sell",
"type": "hive::protocol::asset"
},
{
"name": "min_to_receive",
"type": "hive::protocol::asset"
},
{
"name": "fill_or_kill",
"type": "bool"
},
{
"name": "expiration",
"type": "fc::time_point_sec"
}
]
},
{
"name": "limit_order_create2_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "orderid",
"type": "uint32_t"
},
{
"name": "amount_to_sell",
"type": "hive::protocol::asset"
},
{
"name": "fill_or_kill",
"type": "bool"
},
{
"name": "exchange_rate",
"type": "hive::protocol::price"
},
{
"name": "expiration",
"type": "fc::time_point_sec"
}
]
},
{
"name": "limit_order_cancel_operation",
"fields": [
{
"name": "owner",
"type": "hive::protocol::account_name_type"
},
{
"name": "orderid",
"type": "uint32_t"
}
]
},
{
"name": "pow_operation",
"fields": [
{
"name": "worker_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "block_id",
"type": "hive::protocol::block_id_type"
},
{
"name": "nonce",
"type": "uint64_t"
},
{
"name": "work",
"type": "hive::protocol::pow"
},
{
"name": "props",
"type": "hive::protocol::legacy_chain_properties"
}
]
},
{
"name": "pow2_operation",
"fields": [
{
"name": "work",
"type": "hive::protocol::pow2_work"
},
{
"name": "new_owner_key",
"type": "optional<hive::protocol::public_key_type>"
},
{
"name": "props",
"type": "hive::protocol::legacy_chain_properties"
}
]
},
{
"name": "request_account_recovery_operation",
"fields": [
{
"name": "recovery_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "account_to_recover",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_owner_authority",
"type": "hive::protocol::authority"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "recover_account_operation",
"fields": [
{
"name": "account_to_recover",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_owner_authority",
"type": "hive::protocol::authority"
},
{
"name": "recent_owner_authority",
"type": "hive::protocol::authority"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "reset_account_operation",
"fields": [
{
"name": "reset_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "account_to_reset",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_owner_authority",
"type": "hive::protocol::authority"
}
]
},
{
"name": "set_reset_account_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "current_reset_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "reset_account",
"type": "hive::protocol::account_name_type"
}
]
},
{
"name": "change_recovery_account_operation",
"fields": [
{
"name": "account_to_recover",
"type": "hive::protocol::account_name_type"
},
{
"name": "new_recovery_account",
"type": "hive::protocol::account_name_type"
},
{
"name": "extensions",
"type": "hive::protocol::extensions_type"
}
]
},
{
"name": "transfer_to_savings_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "amount",
"type": "hive::protocol::asset"
},
{
"name": "memo",
"type": "std::string"
}
]
},
{
"name": "transfer_from_savings_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "request_id",
"type": "uint32_t"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "amount",
"type": "hive::protocol::asset"
},
{
"name": "memo",
"type": "std::string"
}
]
},
{
"name": "cancel_transfer_from_savings_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "request_id",
"type": "uint32_t"
}
]
},
{
"name": "decline_voting_rights_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "decline",
"type": "bool"
}
]
},
{
"name": "claim_reward_balance_operation",
"fields": [
{
"name": "account",
"type": "hive::protocol::account_name_type"
},
{
"name": "reward_hive",
"type": "hive::protocol::asset"
},
{
"name": "reward_hbd",
"type": "hive::protocol::asset"
},
{
"name": "reward_vests",
"type": "hive::protocol::asset"
}
]
},
{
"name": "delegate_vesting_shares_operation",
"fields": [
{
"name": "delegator",
"type": "hive::protocol::account_name_type"
},
{
"name": "delegatee",
"type": "hive::protocol::account_name_type"
},
{
"name": "vesting_shares",
"type": "hive::protocol::asset"
}
]
},
{
"name": "recurrent_transfer_operation",
"fields": [
{
"name": "from",
"type": "hive::protocol::account_name_type"
},
{
"name": "to",
"type": "hive::protocol::account_name_type"
},
{
"name": "amount",
"type": "hive::protocol::asset"
},
{
"name": "memo",
"type": "std::string"
},
{
"name": "recurrence",
"type": "uint16_t"
},
{
"name": "executions",
"type": "uint16_t"
},
{
"name": "extensions",
"type": "hive::protocol::recurrent_transfer_extensions_type"
}
]
},
{
"name": "witness_block_approve_operation",
"fields": [
{
"name": "witness",
"type": "hive::protocol::account_name_type"
},
{
"name": "block_id",
"type": "hive::protocol::block_id_type"
}
]
}
]
Back to the original problem
So how did we make our packing problem smaller by doing this?
Well to find our let's run a simple grep and sort command:
grep type out.json |sort|uniq
The result of this command shows us 27 lines.
"type": "bool"
"type": "fc::time_point_sec"
"type": "flat_map<std::string, vector>"
"type": "flat_set<hive::protocol::account_name_type>"
"type": "hive::protocol::account_name_type"
"type": "hive::protocol::asset"
"type": "hive::protocol::authority"
"type": "hive::protocol::block_id_type"
"type": "hive::protocol::comment_options_extensions_type"
"type": "hive::protocol::custom_id_type"
"type": "hive::protocol::extensions_type"
"type": "hive::protocol::json_string"
"type": "hive::protocol::legacy_chain_properties"
"type": "hive::protocol::pow"
"type": "hive::protocol::pow2_work"
"type": "hive::protocol::price"
"type": "hive::protocol::public_key_type"
"type": "hive::protocol::recurrent_transfer_extensions_type"
"type": "int16_t"
"type": "optional<hive::protocol::authority>"
"type": "optional<hive::protocol::public_key_type>"
"type": "std::string"
"type": "uint16_t"
"type": "uint32_t"
"type": "uint64_t"
"type": "vector"
"type": "vector<hive::protocol::authority>"
So now the next thing we need to do is figure out how to pack these 27 (or possibly less) types. This might be a little more involved to automate and might possibly run into the size problems we spoke about earlier because the packing code seems to be template code that needs to be instantiated in order to further process. I'm going to look into this, and in the worst case there will end up being some limited amount of manual work, but we'll find that out when we get to it.
conclusions
I've got a lot of more work to do tu use the generated JSON in a usefull way, but I hope this post has demonstrated how usefull clang AST extraction feature can be when trying to write other language versions of C++ code and automating some of the otherwise manual work. I'm not sure yet if my own particular usecase can be extended far enough to not have to do any manual work on a new hardfork, probably not, but at least it reduces the manual work to something quite small.
I know it is still a lot of effort for a minimal bit of latency reduction, but that really isn't the point of this exercise. The latency reduction is going to be a nice bonus if this succeeds. The prime advantage will be a queryable python API using the overloading features that Python provides.
I'm however first going to explore the packing because its the most chalanging, and if the packing works, the rest will come for free.