I think that effect you mean in portrait shots is probably bokeh, where you get a nice blur on the background but the person is in perfect focus?
In terms of camera recommendations, it's difficult to say because everyone's needs are slightly different. I would suggest looking at an interchangeable lens system (as opposed to 'bridge' cameras which you can't change the lens on) because the range of different lenses can make a huge difference to your shots. For example, there are lenses specifically designed for macro and portrait shots.
As to which system...this is where you have to weigh up cost vs size vs ability. If you are stepping up to a more professional cameras from a smartphone then size will probably be important. The full frame gear like my Canon EOS 5D is expensive, and it can feel very heavy and clunky, although that comes with amazing image quality.
The newer mirrorless technology is starting to become really popular, and they cram some amazing capability into a slightly smaller package than a DSLR. I have just bought a Fujifilm X-T3 and a couple of lenses, and I can't speak highly enough of it. For the size and price the quality is amazing, and it looks beautiful :) I will write a post on what the considerations were for buying it - maybe that might be helpful for you.
RE: Photography Hacks: 5 Tips to Help You Take Sharper Landscape Photos