Hello Steemians, in an effort to keep you informed, we are continuing to provide updates about the different projects we’re working on. Our engineering teams follow a 2-week-sprint cadence, so some of our updates will be more substantive than others. You can view our last engineering update here.
Condenser/Wallet Split
We are nearly done with Stage 1 of the Condenser split. A wallet-only app will be launched at a separate domain. On the Social app we will implement a warning dialog for when the user attempts to log in with anything other than their Posting Key. This is in preparation for restrictions that will coincide with the removal of wallet functionality from the Social app and will disallow logging in with the Master Password entirely. To learn more about your Private Keys and how to use them properly, check out our recent post.
There is additional work to be done after this, specifically relating to improving user experience and security, but when the final stage goes live it will be extremely exciting as the two dedicated apps will be more secure and cost much less to run than the old Condenser while enabling us to add new functionalities more rapidly.
Cost Reductions
We're still on the path to adjusting instance sizes and tuning auto scaling for all applications. In the process of performing this task for Jussi we discovered that a relatively large portion of our data transfer costs are coming from inefficient 3rd party applications and bots. We have a short term plan that will mitigate the worst offenders and then we can continue adjusting instance sizes and autoscaling for Jussi along with the rest of our applications. These changes should not impact 99% of applications that utilize api.steemit.com, and the few that it does can likely solve their problem by requesting data that does not change frequently.
Mira/RocksDB
We would like to share a little more about the process that we are using to optimize Mira. We are iteratively profiling the steemd reindex to identify slow spots that we can further optimize. When we started this process, we were only reindexing the first 500,000 blocks and that took several minutes. We were able to optimize some low hanging fruit which has allowed us to reindex more of the blockchain and profile that performance. The tricky thing that we are running into now is that some performance optimizations result in increased performance over some blocks, but a decrease for others because the workload of the blockchain is not static.
A great example of this is a slowdown that we observed last week around block 19,000,000. For whatever reason, reindex performance ground to a halt and was averaging one block every 1.15 seconds. That is not acceptable for live performance, and is horrendous while reindexing. The volume and/or types of operations happening on the blockchain changed around that time which resulted in a degradation of performance. Upon investigation we found that some common queries such as, "What is the next comment that is going to be rewarded?" were taking far too long and consuming nearly all of our runtime. We decided to implement a system that caches the first key in an index so that queries like the one above are quicker to answer. You can find the commit that introduced this optimization here.
When we reindexed with that change, we got to block 19 million in record time for Mira, (18 hours) and have been able to progress beyond block 19 million at a rate of approximately one block every 0.012s! Previously we were reindexing up to blocks 500,000 and 2,000,000. With this change, our reindex time to 500,000 was slightly slower, if it changed at all. For 2,000,000 blocks it represented an increase of less than 1%, but for the longer reindex we have observed a performance increase of over 9000%!
This outlines the primary challenge still ahead of us as we continue to optimize Mira. Because our largest gains are only realized on longer reindex times, there is an inherent latency in identifying and optimizing slow downs. We have a small handful of problems that we are actively working to optimize while we continue to identify the next targets for optimization. While there is still much work to be done, we are getting tantalizingly close to running a live steemd node on Mira.
As you can see from this update, one of the unique challenges we face as a team is that our attention is divided between cutting-edge blockchain work, and cutting-edge front-end development work. Most, if not all, of these challenges are a consequence of the unparalleled traffic (in the blockchain sector) that is generated by steemit.com and the ecosystem of Steem apps. While this certainly makes our jobs more challenging, we believe it is our willingness to tackle these challenges head on that has enabled us to remain years ahead of the competition.
Be sure to follow @steemitblog if you would like to see more engineering updates like these!
The Steemit Team