It pretty much depends on how much you're willing to pay for starters. Every time someone asks that exact question, they get bombarded by "GCW-Zero this, "Android that".. . Which there is nothing wrong with suggesting as they are for the most part impressive tech.
But everyone likes to assume that the new guy has $150+ to plunk down on or hours and hours of time to sit down and troubleshoot the devices.
What I would recommend is put that powerful PSP to use as it has well over a decade of homebrew support as it is and offers many if not all of the systems you mentioned. It would also be great practice for everything else that's out there in the homebrew / open-source scene.
The DS also has flash cartridges with tons of homebrew as well although it is much more limited in scope compared to the PSP.
If you're pretty much done with those consoles, then yes, the first thing I would recommend is to decide how much you're willing to spend, read some reviews (real reviews, not fanboy advertising, cough-cough) i.e. youtube videos, etc. and go from there.
Personally I get along fine without breaking the bank for the top-notch, latest and greatest handhelds and I'm happy with Neo Geo Arcade, CPS-1, CPS-2 and everything below the PSX.
I enjoy emulation on PSP-3000, 3DS-XL, Gemei/Dingoo A330, Dingoo A320, Game Boy Advance-SP, Hell, even Game Boy Color! haha!
What I like about these is there low price point, easy, drag and drop setups, simple trouble-shooting, solid builds, robust software support, etc, etc.
Seriously though, take your time and think about it.
Last thing you want is to regret dropping almost $200 on a console with a crap d-pad and long nights of ripping your hair out troubleshooting. 3-)