In understanding how human beings are motivated to cooperate it is important to be aware of social exchange theory. The video below is a very good introduction to social exchange theory:
The video above is the best video I've found on describing social exchange theory.
Worth = rewards - cost
Simple social exchange models assume that rewards and costs drive relationship decisions.[14] Both parties in a social exchange take responsibility for one another and depend on each other. The elements of relational life include:
Costs are the elements of relational life that have negative value to a person, such as the effort put into a relationship and the negatives of a partner.[16] (Costs can be time, money, effort etc.)
Rewards are the elements of a relationship that have positive value. (Rewards can be sense of acceptance, support, and companionship etc.)
As with everything dealing with the social exchange theory, it has as its outcome satisfaction and dependence of relationships. The social-exchange perspective argues that people calculate the overall worth of a particular relationship by subtracting its costs from the rewards it provides.[17]
Worth = Rewards − Costs
Game theory is important when thinking of incentive structures
The question, why would two complete strangers who don't care at all about each other cooperate? If we are trying to get these two complete strangers who don't care about each other to cooperate then we must understand game theory. Cooperation can be understood by understanding the iterative prisoner's dilemma:
Over a series of games the players can learn to cooperate over time. They work together because they think they can get more from cooperation over a long period of time than by defecting over the short period of time. The consequence for being a defector in a game could be that the other player defects in the next game because they hold a grudge. This is called the tit for tat strategy, and this ultimately is what leads to cooperation as a way to avoid being defected on.
What does this mean in the context of a smart contract?
Cooperation is promoted by punishing defecting. So by having the incentives aligned so that the easiest way to win is to cooperate (easiest as in least risky) then typically the participants will cooperate for the easy wins. In a smart contract a scammer would simply lose their security deposit if their reputation score is too low. This would put risk so that they have something to lose when they scam. People can trust the merchant who has something to lose, and in general the more a person has to lose the more trust can be quantified because they are the one's taking a risk equal to whomever is trusting them.
So this brings us to Proof of Stake, and in situations where a person can have a guaranteed income possibly for many years into the future, would they become greedy and risk it all for control of the network? If they determine the cost of this is too high then they'll never even risk it and we can trust them. Casper is interesting because it works on something like this where you can lose your tokens of value if you deviate from the protocol and that acts as a sort of punishment for defecting. Reputation also has a similar mechanism if done right, and can act as a punishment for defecting where reputation is damaged (arguably reputation is more valuable than money).
This blog post is just a simple overview and stream of thought on basic concepts. In general, this is the basic knowledge you would need to design a smart contract which considers human motivations. Humans aren't only motivated by money, and in the math you would want to think of the rewards as "anything of value" to the participant receiving it and the punishment as "removal of the thing of value". It just so happens that Ethereum offers tokens which are worth money, but this can be praise, it can be up and down votes, it can be reputation, it can be attention, it can be respect, it can be anything of value which can be quantified on a blockchain.
In my opinion the security of distributed information systems of this type, depend almost entirely on the mechanism design, which is based on either economic assumptions, and or the game theoretic assumptions. This is the case for any Proof of Stake or Proof of Work, where all participants can be modeled (and should be) in simulation form, and the game theory can be analyzed.