Back when the GBA came out, I thought I was hot shit because I was the only person I knew that was even aware of flash cartridges. Flash carts are rewritable cartridges for game systems that let you play roms for that system on the actual hardware, but also run homebrew.
I remember blowing somebody's mind, showing them a Windows-like OS on my GBA from which I could launch GBA roms, or roms for NES, Master System, PC Engine and so on. But then the GP32 came out, and it was my turn for a head explosion.
The GP32 (pictured above), released by Korean company "Gamepark" in 2001, was basically the guts of a cheap pocket PC running Linux but in gaming handheld form factor. It used Smartmedia cards which were popular at the time to store software. The screen was larger than the GBA's and higher resolution (320x240 instead of 240x160).
Besides running homebrew and emulators, it had its own limited selection of games which ran natively and were developed by Gamepark themselves. Many of these were later ported to PSP, such as Astonisha Story R.
Not content to rest on their laurels, Gamepark then came out with the GP2X in 2005. It improved on the GP32 with a backlit screen (the GP32 screen was transflective, like the original GBA) and more horsepower. It also used SD cards, which by that time had become the clear winner of the flash storage format war.
GP2X could manage things the GP32 couldn't, such as fullspeed SNES and Genesis emulation, even Neo Geo. It could not manage PS1 however. That would have to wait a while.
The GP2X F-200 variant was released in 2007, the only major change being the addition of a touchscreen to compete with the Nintendo DS. However this feature could also be used to imitate the mouse when emulating old computers, such as the Amiga line.
With the release of the GP2X Wiz in 2009, finally PS1 emulation was possible, albeit not at full speed for most games. 2D titles ran fine, but 3D titles lagged noticeably.
The dual D-pad design was uniquely suited to games like Robotron 2084 and Smash TV. The build quality was also improved, and it was the first Gamepark handheld with a built in lithium battery instead of using AAs.
The same year, Dingoo Digital (a competitor) released the Dingo A380. Based on Nokia cell phone hardware, it boasted some impressive 3D capabilities for the time, as seen in the included horror title "7 Days Salvation". It was up to emulating everything up to SNES, though just barely.
It's hard to get ahold of now since it's no longer made and is still popular as a cheap but solid entry level emulation handheld. It is still not a bad choice if you're unsure if you want/need such a thing and don't want to blow too much $$$ to find out.
The Caanoo came out one year later, in 2010. It was Gamepark's last handheld and by far its most powerful, including all the advances from prior handhelds. Touch screen, integrated lithium battery, the works. It finally had the 3D hardware necessary to run PS1 games full speed, but could not manage N64 or Dreamcast competently due to lack of an FPU.
That same year, the Pandora came out. The first kickstarted handheld to my knowledge, it boasted immensely more graphical horsepower and could handle N64 with ease. It could also emulate Dreamcast, albeit not as well. The main draw of the Pandora was the variety of features.
It has dual analog thumbsticks, a full keyboard, a touchscreen, dual SD slots, wifi, and everything else you could ask for in such a device. It was after all developed by Gamepark fans who felt frustrated by the limitations of Gamepark handhelds.
The GCW Zero, released in 2013, is another handheld kickstarted by fans of the Gamepark handhelds. Both GCW Zero and Pandora sought to fill the void after it was revealed that Caanoo would be Gamepark's last handheld gaming console. There's not much to say about it, other than that it does the job and is well made.
The Dragonbox Pyra is yet another kickstarted handheld and successor of the Pandora, released in 2015. It emulates Dreamcast full speed and anything older than that. Unless you're looking to emulate Gamecube or Wii, this bad boy will handle anything you throw at it, including source ports of popular 3D shooters from a few years ago.
JXD is difficult to fit into this chronology because they have released so many devices. All of them are just Android tablets with game controls built in. Some are clamshell 3DS imitators. Others, like the one pictured, resemble a giant PSP. Being Android based, these are remarkably powerful and can emulate Dreamcast with ease. The very newest ones can even manage Gamecube and Wii emulation.
But then, if you already have a smartphone, odds are good it's more powerful than most of the devices on this list. Economy of scale has brought laptop grade hardware, crammed into smartphone form factor, down to a price most people in the first world can afford.
There's any number of clip-on game controllers you can use to add physical controls to your phone, not just for emulators but native Android games, the quality of which is remarkably good these days. Personally I prefer to prop up my phone or tablet like a little TV and use an Xbox One S controller separately, as you might play the Nintendo Switch, but it's up to you.