Support Proposal 303 on PeakD
Vote for Brianoflondon's Witness KeyChain or HiveSigner
This is a value for value post: see the explanation in the footer.
V4V.app was hacked
On Friday 11th October my systems alerted me to something strange going on with v4v.app. A bunch of outgoing payments didn't seem to match up with the proper notifications from my back end.
I had a bad feeling so I jumped out of bed and very quickly shut down the API server and the Lightning node and started looking at log files.
Toward the end of this post I'm going to tell you later the broad type of attack which caused me to loose Bitcoin but not the very specific details.
Logs
Fortunately I keep quite good logs and I could see I had a problem quite quickly. I don't want to explain exactly how this happened in public but this was a genuine security failure on my part.
The problem was I did not think like a hacker when I built the KeepSats functionality which I introduced a bit too quickly earlier this year.
Taking the site down
Friday morning was a particularly bad time for this to happen. This year the Jewish holy day of Yom Kippur started on that Friday evening and I knew I did not want to rush fixing the site. I took the difficult decision to turn off most aspects of v4v.app. I would not be able to work on Yom Kippur.
Fixing the site
I set to work on Sunday morning fixing things. I worked on this all day Sunday and put in place some changes which I thought might take care of the problem. Unfortunately this first set of fixes wasn't quite good enough. It slowed the rate at which the site could be exploited but the hacker came back and was able to extract a few more sats before I stopped him (satisfyingly I did stop him just after he deposited 2m sats on the site and before those could leave).
The next few days I spent re-writing a significant part of the API system which is public and which my v4v.app website uses behind the scenes. That was where the vulnerability lay.
The Problem
The broad outline of the problem was that Lighting invoices take a few seconds to pay and in special cases can be set to take many minutes or even 24 hours to pay. Once my Lightning Node has started to pay one of these invoices, there's no way to interrupt the process.
Double Spend
The mistake in my code was that I wasn't properly deducting an outgoing payment from a user's balance whilst this payment process was in progress. The attacker first deposited some sats on my system with the KeepSats function. The attacker then used custom code or curl commands to send multiple, simultaneous Lightning invoice payment requests. My system jumped into action and started using the same balance to pay multiple outgoing invoices. He would send in 1M sats and then get 3M out. This happened over the course of a few minutes until I shut everything down.
The Loss
In the end I lost somewhere around 0.13 BTC or 13m sats. I'm not even going to do the maths on the dollar value, and this was a few weeks before the current massive highs we're reaching now in the BTC price.
Fixed?
I hope it is fully fixed now. The answer lay in imposing very strict filtering on the API system which pays invoices around timing and simultaneous requests. This has no impact whatsoever on anyone using the v4v.app front end, but it should block any abuse behind the scenes using custom code to copy what my front end site does.
What this means for @v4vapp
It means that the @v4vapp system has cost me much more to operate than the total amount of the very low fees the system has collected up to now. The total fees I've collected are in the 250,000 sat range so a loss of 13,000,000 sats is clearly well beyond that.
On the other side of the ledger is the DHF so I have received enough to swallow this loss. Mentally I'm looking at this as a security audit fee. I've done nearly all the dev work on this project myself paying out almost nothing to others. This was an unwelcome expense, but that's how I have to view it.
Value for Value
For the last few months while building @v4vapp I was generously supported by the DHF. Going forward I have a much more modest support which covers direct server costs and a little of my time.
If you appreciate the work I do on and around Hive, you can express this directly: upvoting posts on Hive is great. Also consider a direct donation (there's a Tip button on Hive or a Lightning Address) on all my posts.
Support Proposal 303 on PeakD
Support Proposal 303 with Hivesigner
Support Proposal 303 on Ecency
Vote for Brianoflondon's Witness KeyChain or HiveSigner
