The slowdown does not appear in your case because 20fps is really low. If you render at 60fps, you will render three times more frames in a second, so the slowdown will be three times higher. Probably more than that actually, because you eat more memory baudwidth.
If you really want to see the difference, try to implement it whithin gpmark.