We all know that there are a bunch of idiots everywhere. When something is going well, there would always be some negative energy from different sides. From incidents like these, there are always frustrations and learning. For the past few days, the Hive Engine public nodes have been suffering from multiple attacks. I thought it would be better if I give an update on what is happening. Some of my friends asked me what was going on with Hive Engine nodes.
From the looks of it, it was a planned DDoS attack on the public nodes. The list of all available Hive Engine witnesses are available on this page. The website also displays the IP address of the accounts running the witness node. This is quite common and the problem is not because of displaying the IP address. The main problem is that someone decided to attack the Hive Engine nodes with a bad set of queries. Until yesterday, I was not sure if it was an intentional attack or not. But when it happened a couple of times in the last few days, we found out that it was an intentional attack on the public nodes.
In the last few days, many Hive Engine witnesses had their servers running at 100% capacity. You can take a look at the below image to see how all the CPU cores are at 100%. It was a nightmare to run the servers at 100% capacity. It was not a few nodes that were affected. Several nodes including the backup nodes were affected because of this and people had to block the RPC endpoint on the node finally to stop being attacked.
I thought it was an unintentional malformed query that was creating this issue but later we found out that this attack was intentional. If you look at the above two screenshots, the querying happened continuously on multiple nodes at the same time with several IP addresses. It was very hard to track these IP addresses and block them individually.
The initial solution was to introduce rate limiting. Some people introduced a rate limiting on the domain level using Cloudflare, some introduced it on the web server level on nginx and some introduced it directed in the code with the help of express-ratelimiter package. This helped to some extent to block attacks from the same IP address. As you can see above, the attacker was clever and wanted to do the attack from multiple IP addresses. This always kept the CPU percentage at 100%.
The above screenshot was during the peak time when the server was fully loaded. Mostly this was because the Mongo server was very busy trying to serve all the requests from the attacker. The below image was from another powerful server that I had with 12 cores and that also had the same problem.
The server was literally struggling to serve all these requests during the attack. The final solution that worked was to just block the RPC port from the public and enable it only for specific IP addresses. I have associated a domain to some of the nodes that I manage and these domains are behind Cloudflare with a rate-limiting set. On the other nodes, I'm just keeping it private because other services that are running on these hardware are getting affected because of this issue.
Today morning the attacker was again back with another attack. This time the issue was not very big as many nodes have already added rate limiting and some nodes have blocked their RPC port from public access. In the message on the request, the attacker had mentioned: "You Win For Now". This was the funniest thing and this actually confirmed that it was an attack.
Finally, the great thing that happened out of this event is that many great minds worked together on this issue and they have come up with several solutions to handle this problem. Some of these solutions are experimental for now and would soon be incorporated as standard operating procedures. I had a great feeling that "Necessity is the mother of invention" and that way thanks to the attacker. We had to waste a lot of time on this but it was indeed a great learning.
If you like what I'm doing on Hive, you can vote me as a witness with the links below.
 |
 |
 |
 |
 |
 |