As I wait for the slow sync to finish, I think I came up with a clean solution for the RT block handling. The problem as discussed before is transitioning from using one set of data files to another, seamlessly and possibly under API load.
Sometimes the best solution simply avoids the difficult case entirely and that is what I propose doing.
Since iguana doesnt use any database, it avoids the problem of getting a corrupted database requiring to regenerate it from scratch. Of course, any of the various files can get corrupted, but with the vast majority of data in permanent never changing files, it is possible to write protect them, or even put them in to a read-only compressed volume. The ~5% of files that have volatile data is further divided into a write-once dataset and a truly changing dataset.
It turns out there are just a couple of fields that have to change and it is above the blockchain abstraction layer. This makes sense as the blockchain is supposed to be immutable! And it means much less expensive storage can be used to store 95%+ of the data.
However due to the fact that any specific datafile might be missing or corrupted, iguana does an init time reconstruction of the required data to make sure that it is all there and if not, it tries to generate the missing files. And the brand new state of right after install uses the same logic to start the parallel sync. It has taken many months to get it so most of the time it properly detects and (re)generates what is needed, but this is complex enough from a static starting state to achieve.
The problem of then modifying the state, generating new files on-the-fly while it is being used and able to detect all cases, well I tried for quite a while and all attempts failed some test or another. So, my latest idea is to simply avoid this case.
Like the doctor says, "if it hurts, dont do it"
In this context, what this means is to start from a valid initial state and maintain correctness by adding (reorging) the RT (realtime) blocks. That's it. yes, it does sound not so easy just limited to that, but what is missing is the dynamic creation and substituting updated bundle files, spend vectors, balance files and ledgers. My blind spot has been that all access to the iguana data has been bundle based, so it needs a specific bundle to query. A bundle has a fixed number of blocks and there are too many dependencies on this assumption.
Instead of creating a RT bundle that expands until it fills up and making a new one, swapping the edge of the dataset, I just need to have an open ended RT dataset that isnt bundle based. This will require modifying the places that makes queries about the blockchain, but it avoids the really difficult part and I already have special casing for querying the RT bundle approach. A few additional cases where that virtualization was relied on will need to be changed to support the extensible RT dataset, but nothing too hard to do.
This does mean that until the iguana is reset, it will be a bit behind on bundle files and latest ledger, however an old ledger + all tx since the ledger start has to match a later ledger + all tx since the later ledger's start, so the important part of having the same data will be retained.
James