First, two caveats.
One, there is no single libertarianism that all self-proclaimed libertarians will agree to. This is suggestive rather than conclusive.
Second, the current Libertarian Party has been co-opted by people who are best understood as paleoconservatives (I'll explain at the end).*
Libertarians are, at core, skeptical about people having power over others. They are, therefore, skeptical of government, for which power over others is the foundational concept.
What about businesses, people ask? Firms can hire and fire. But, also, employees can pick, choose, and walk. And they do. (And doing so causes costs to the firms, who have to search and train new employees.) The business world is one of mutual agreement - one party withdrawing their mutual agreement is not the same as force, even though it can affect the other party negatively (divorce, for example, is not force).
Government is not based on mutual agreement - social contract theory is false. Voting in an election does not mean you consent to the policies of the victors - and don't pretend that you actually do; instead you vow to continue fighting them.
Libertarianism does not depend on the basic goodness of people. If we thought people were inherently good and trustworthy, we wouldn't be so worried about some having power - we'd assume they'd do good things with it. Instead, we assume that people having a great capacity for venality, we don't trust them with power over others.
Even the better sort are likely to do more harm than good, because society and markets are complex adaptive things, far too complex to be properly managed from the top down. The best laid plans, the road to hell, etc. We all know good plans can go badly, and the more complex the system you're dealing with, the easier it is to colossally fuck up. As a case in point, poverty was declining across the years until Johnson's Great Society programs and his "war" on poverty - it didn't really being to decline again until Bill Clinton's welfare reform, which was falsely predicted to throw many more people into poverty.
What about markets? Isn't there too much power in markets? The more competitive the market, the less power there is. Ironically, perhaps, the more fiercely firms have to compete against each other, the more they have to cooperate with their customers, to keep them from defecting to competitors. Don't believe it? Consider your own consumer choices, and how you try to avoid being suckered.
So do you believe businessmen are more moral than politicians? Some libertarians do, but I don't, and it's not necessary to do so. Assume they're equally moral or immoral - the key is that they face different incentives.
Yes, businessmen have incentives to lie and steal...until the customers find out they've been screwed over. Politicians also have incentives to lie (and to steal in legal ways - it's amazing how many people become millionaires once they're in Congress), but how often do people stop supporting their party's nominee because s/he lies?
In the market, we seek out the supplier who best meets our needs for managing our own lives. In politics, we seek out the policy supplier who promises us they'll manage others' lives to our own satisfaction. Libertarians don't respect your desire to control others. Managing our own lives is tough enough - none of us is qualified to manage others.
It's not that libertarians don't care about people. It's precisely because they do care - because they see everyone as of equal intrinsic worth - that they want to limit how much some are controlled - dominated - by others.
That's why they prefer open, or at least loose, borders - it gives millions of people the chance to better their lives. It's why they advocate for more competitive markets - competition drives down prices and drives up quality, benefiting consumers (in fact libertarians are not pro-business, but pro-consumer). It's why they advocate making it much easier to build housing - because increased supply drives down prices, making it more affordable.
Libertarians have often, unfortunately, used the Gadsden flag, with the snake and the slogan "Don't tread on me." The proper libertarian slogan is "Don't tread on anybody."
*Paleo-conservatives, as best I can tell, are most interested in their freedom to discriminate, generally on ethnic grounds. Good libertarians will support that right (freedom of association), but without respecting the bigots (just as the ACLU defended the right of Nazis to march in Skokie without respecting the Nazis themselves).
And specifically, paleos want to use the power of government to limit immigration, again primarily based on ethnicity. That's not only using government to coerce people but doesn't view people as all intrinsically equal.
Many of these people are affiliated with the Ludwig von Mises Caucus of the Libertarian Party. But don't blame the deceased Mises for that - he didn't share their despicable views.