If you undo the changes It should be ok.
Volume button use a different input device (keyboard) as normal controls (ADC driver inside kernel) so not sure if what you wanna do is possible. Maybe with tincore keymapper you can without editing kl files which is generally a bad idea becausw those files get corrupted easily (even with correct permisssions) you can also try editing kl files for ps3 and Xbox gamepad adding the lines you need for volumen buttons, but I doubt that Will work unless joystick driver is rewritten. ( You have nothing to lose).
I don:t know what problem you have on ps3 mode. You can rmap buttons in any emulator so It doesn't matter Too much if you use ps3 or Xbox mode. Also for retroarch ps3 mode is must. Null joystick disable buttons, so that's a normal behaviour. Null is only needed for android games that have bad joystick Support.
Basically try undoing the changes you have done and make sure toundon't change permisssions of the kl files. Or try with tincore. If not, the only option is to reinstall or copy the original kl file from someone with the rom installed. No matter if it's my ROM or stock because controls are the same on both. Also make sure you don't have gpd game mapper active just in case.
There is not great solution for l3 and r3 on q9, since you will lose volume buttons. They should have included Analog buttons like in g7.
I suppose you tried this guide:http://boards.dingoonity.org/gpd-android-devices/(gpd-q9)-how-to-remap-the-volume-buttons-to-l3-r3/
I guess you have done the same thing so It should work. Make sure to double check permisssions. Also maybe that change is not working ok on Xbox mode, or It needs another kl editing for Xbox mode. I personally have always used ps3 mode, never needed to use Xbox mode for anything. Also, joysticks modes and drivers are encoded in ADC driver inside kernel, so maybe further changes are needed.