Possibly, but again, if someone comes to a house with a gun to throw them in a cage because they didn't give a portion of their income to rulers, and only 10% percent of the population agrees with the validity of that person's "authority" to cage them, don't you think we'd see more people with guns saying, "Uh, nope. Nice try, but you have no authority here. I'll freely defend myself against you."?
The numbers I'm talking about would, I think, have a direct impact on the "belief in their authority" concept. I wrote about that here as well.
Yes, people may come after you, but there's also this "social contract" idea and the "consent of the governed" that holds it all together. We've seen numerous examples throughout history where humans stood up against obvious injustice, even to the point of personal harm, when their position is the morally correct one.
RE: A mathematical explanation how voting third party in 2016 is the only vote that counts