A Rough Patch
My witness node crashed about 1.5 weeks into the month and needed to be reindexed. I had chosen the server provider, Privex, because of the promise of being able to get back up and running more quickly with a server reset being that it includes a recent snapshot of the blockchain.
After resetting the server and spinning up the witness again, I was synced up to around August of this year. I had hoped the snapshot would have been more recent, but I thought everything would be fine and it would catch up quickly. Almost a week later, it was still not caught up to head. I contacted @rishi556 who does support for Privex to see if he had any suggestions, and he was very helpful as we attempted to troubleshoot the delay. During this process, it was determined that perhaps SSDs can no longer keep up with the needs of running a Hive node, so that tier of service was removed from their offerings.
Meanwhile, I went ahead and purchased another Privex server located here on the west coast with an NVMe drive configuration. The hope was that this would sync up within hours. So I got the new server configured and running, and the snapshot started from July of 2023. I wasn't worried because it seemed like it was going to catch up quickly. Unfortunately it was not the case, and 24 hours later it has barely moved a week. I ended up cancelling the new server.
I Built My Own
I decided to go ahead and put together my own machine to run my witness node on. I have been a system builder for a local 3D house for years and have computer parts coming out of my ears, so it was not a big deal. I also have several rackmount server cases and power supplies from my crypto mining days. The server has Threadripper processor, 128GB of RAM and a couple SABRENT 2TB Rocket 4 Plus NVMe 4.0 Gen4 drives. I originally put 64GB in there, but found a bigger kit in a box of parts, so went with it. I was not going to have any bottlenecks this time!
I set up the node quickly using @someguy123's Hive Node in a Box and went from nothing to fully synced in about 26 hours. I went ahead and switched block producing to this node and a few hours later, my original node finally caught up.
Current Situation
As it stands right now, I have renewed my original Privex node for another month with the SSDs and will be using it as a backup for for the one I am running in my work server closet. I don't want to be caught in a situation again where I am down for weeks! With a power or internet outage I can be back up and running in minutes rather than days.
I have been running stable and producing blocks for well over a week now and can quickly switch to the backup in case of emergencies.
Ranking Dropped
Unfortunately, I did not gain any support this month - but didn't lose any either. However, I did drop from 89th to 91st place. This was not due to losing votes, but rather a result of a couple of witness nodes gaining more votes than me. On the upside, had I not been down for so long, I would have finally broken even on my costs!
It was a hard lesson learned, but I definitely feel in a better position to recover quickly the next time something like this happens. It is always better to have a backup node! I also learned that it isn't necessarily faster to download a block log to replay from. Even with a gigabit+ internet connection it can take days to download as the file transfer rate may be throttled.
I would really appreciate your support!
