Under a 100$, the PSP is a good choice i would say yeah.
You can easily put a CFW on it, overclock the proc, and you now have access to Retroarch.
(including Mednafen-PCE fast, among others...)
It can emulate the CPS-1, CPS-2 and the Neo Geo but as far Arcade goes, it is not powerful enough.
(even the PS Vita struggles with some Arcade games)
And that's pretty much it...
Everything else either requires modding (a modded RPI3), costs more than 100$ (PS Vita, SHIELD) or it is just not worth it. (a cheap android device + MOGA controller)