I think most people don't see all the scam accounts that have their rewards removed. People are free to DV as they see fit and a few will be nasty about it. I hate to see good people chased away. I think the system got adjusted so that we can give more up- than downvotes.
This site gives some insight into what goes on. I think there are a few long-running feuds.
RE: Thoughts about Downvotes