Which downsides would those be?
I was specifically referring to "hard" redirects, in case that wasn't clear; as well as the reasons mentioned above there's the added hassle of maintaining the redirects. The normal "redirected from" links don't appear across wikis, so you need to approach the page in an indirect manner in order to edit it.
Redirects (soft or not) are advantageous if you're wanting an external page to appear in a local category, but other than that one instance the only actual difference is whether you have to type a couple of extra characters when first adding a link on a page.