Repository
New Project: txJsonRpcQueue
In this post I want to introduce Project txJsonRpcQueue, a highly portable non-API-compatible rewrite of asyncsteem/asyncsteem3 Python STEEM JSON-RPC client lib that aims to run on both Python2 and Python3 and aims to run both under the Twisted asynchronous event library and framework, and in a Twisted free Python3 setting running under asyncio. txJsonRpcQueue Is currently a work in progress.
If you want to program STEEM applications in python, there are multiple libraries to choose from:
- The oficial steem-python library.
- The BEEM library by @holger80
- The LightSteem library by @emrebeyler
- The AsyncSteem library by me.
- The AsyncSteem Python3 port by @scottyeager
Each of the above libraries has its own strengths and weaknesses. None of them is perfect for every type of application, and none of them isn't arguably the top choice for some specific use case. As the author of the original AsyncSteem library, I want to focus on those use cases that favor the use of an asynchronous event framework.
To get a better idea of the type of applications an asynchonous JSON-RPC library for STEEM allows you to write, have a look at this tutorial describing how to build an asyncsteem python web app with micro-transaction-based steem user authentication, Redis and Jinja2
There are currently two versions of AsyncSteem. My version that works exclusively with Pyhton 2 and Twisted, and @scottyeager's version that works exclusively with Python 3 and Twisted. Neither currently supports running with Python3 and asyncio, in the absence of Twisted, and neither currently support signed operations. While, as I lined out in the Twisted tutorial series ( part 1 , part 2 and part 3 ), Twisted and asyncio can be made to play nice in a single application, a significantly large group of Python3 users seem to prefer using asyncio without Twisted in their asynchronous Python3 applications.
Given the desire to have a single code-base for Python 2 and Python 3, and given the call for asyncio-only support, I have looked into merging asyncsteem with asyncsteem3 while also adding asyncio only support and I came to the conclusion that a (low-level) API-compatible merge should prove close to impossible. Not wanting to break the existing code of asyncsteem users, I came to the conclusion the best move would be to declare a feature freeze for the asyncsteem library and start working on a non-low-level-API-compatible rewrite.
In terms of portability, txJsonRPCQueue aims to run:
- On Python 2 with the Twisted asynchronous event framework.
- On Python3 with the Twisted asynchronous event framework.
- On Python 3 with asyncio and the aiohttp library
New hysteresis queue based design
Like asyncsteem, txJsonRPCQueue works internally with a queue of pending commands. While a solid concept for working with batches of commands, something the STEEM APPBASE JSON-RPC API supports, there is the potential of logical errors in program design making the queue grow beyond the limits of the available process memory.
Say for example, you are streaming new blocks. Now when a vote for a post of yours comes in, you want to see if the person voting is connected to you within N levels of separation. Things could easily explode. Your target follows 5000 accounts, each of the account your target follows follow on average 1000 accounts, and before you know it, your single up vote would end up resulting in millions and millions of queued queries, eventually leading to massive delays in your application and in the worst case, your program bailing out when the process runs out of RAM.
The new implementation solves this problem by introducing a more controlled form of failure. The hysteresis queue has a low water and a high water level defined for it. If the number of pending commands reaches the high water mark, any successive command will result in the future or deferred for that command immediately resolving to an exception/error. This behavior will persist until the enough pending commands have been processed for the queue to reach the low water mark.
Queue, Forwarder and Injector
The new low-level API is build around three distinct entities that need to work together. Two of the three are strictly needed, the third is meant for more robustness in the light of API-node unreliability.
- The Wildcard Queue
- The RPC Forwarder
- The Host Injector
The Wildcard Queue provides the user with a deferred (Twisted) or future (asyncio) based hysteresis queue buffered remote procedure call interface. Note that the Wildcard Queue has zero knowledge of the APPBASE STEEM JSON-RPC API. The Wildcard Queue works with the Python getattr trick to create a wildcard interface both on the API choice level and the command and arguments level. The APPBASE API definition works with wildcards, but also allows for a default API to be set, so the following two would be equivalent:
#Use the asyncio/future based implementation of the Wildcard Queue
from txjsonrpcqueue import AioWildcardQueue as WildcardQueue
steem = WildcardQueue(low=8000, high=10000, namespace="condenser_api")
..
#With an explicit namespace
future_one = steem.condenser_api.get_block(12345678)
#Using the default namespace
future_two = steem.get_block(12345678)
Just using the WildCard Queue won't do much. You will be filling the queue, but nothing will ever get processed from it. To let things work as they should, you must instantiate an RPC forwarder bound to your queue. In the simplest form, providing a static API node adress.
...
steem = WildcardQueue(low=8000, high=10000, namespace="condenser_api")
forwarder = RpcForwarder(queue=steem, host_url="https://api.steemit.com")
...
Where hard coding an RPC node could be fine if you are running your own private API node, or have great confidence in the availability and performance of the chosen node. The past though has shown us that the later isn't always justified. To create a solution that allows us to go around that problem without making the core library too dependent on the specifics of STEEM API nodes, the RpcForwarder can be used with an API specific Host Injector. Currently no Host Injectors are implemented yet, but the hooks are there in the core lib to allow their creation.
Roadmap: generic.DNSHealthHostInjector
The below is currently just a plan, based on @holger80's @fullnodeupdate concept. The idea would be that a few of us could run a little server that continuously checks the response times of all known full API nodes, and based on that, as a simple DNS server for a sub domain, resolves TXT record requests for a specific domain name to the currently best performing JSON-RPC node.
For example a TXT query for appbase.steem-api.timelord.ninja could resolve to "steemd.minnowsupportproject.org" at one moment, then resolve to "api.steemit.com" 30 minutes later.
As said, neither the little server nor the Host Injector is currently ready for use, but when they are, imagine the above code changing to something like this:
..
txjsonrpcqueue.generic import DNSHealthHostInjector
..
injector = DNSHealthHostInjector(entry="appbase.steem-api.timelord.ninja")
forwarder = RpcForwarder(queue=steem, host_injector=injector)
Roadmap: asyncsteem.ActiveBlockChain and asyncsteem.DateFinder
While the low level API should be drastically different from the asyncsteem API, there really is no reason not to provide a compatible API for the higher level stuff. The plan is to implement the compatible higher level API so that programs using only the higher level API might swap:
import asyncsteem
with:
from txjsonrpcqueue import asyncsteem
Roadmap: steem.AccountKey
One of the main shortcomings of asyncsteem at the moment if its lack of signing support. The current plan is to use the LightSteem signing code as a reference and implement a second getattr style wildcard interface for signed operations, where the only STEEM API specific code the library knows about is the serialization + signing part. Ideally an Account class should end up providing a wildcard API that should work approximately as follows:
from txjsonrpcqueue.steem import AioAccountKey as AccountKey
..
accountkey = AccountKey(key=my_config["posting_key", queue=steem)
vote_future = accountkey.vote(
voter="mattockfs",
author="pibarabot",
permlink="/stats/@pibarabot/flag-war-stats-for-posts-made-on-2019-01-23",
weight=10000
)
Feedback and/or feature requests.
As said, txJsonRpcQueue is still in the early stages of development. The code available today must be considered early alpha, but with my experience with asyncsteem, things are progressing quite fast. If you have any input you would like to share, especially if you are currently an asyncsteem or asyncsteem user, please comment below.
The API isn't solidified yet, so sugestions about API structuring, etc are still possible. I'm also very much open to feature requests at this point.
How to contribute
If you wish to contribute to the project, please comment below. You are welcome to claim any of the roadmap items above or propose other additions you would like to implement. Other contributions that would be welcome at the moment are code review and just general feedback on the ideas above, the API, and the portability between event frameworks and Python versions.