TL;DR
Click Button ${button} True
Seems you already figured this out, but I had the same problem and it took some time to figure it out. So this is for future reference.
ExtendedSelenium2Library waits for AngularJs to be ready to process the next request. As can be seen here ExtendedSelenium2Library (and like you already said). So by using these commands that wait for Angular to be ready, when you are not using AngularJS, it is necessary for the timeout to kick in which is about the 12-14 seconds you experienced. To still be able to use the extended library and have "fast clicking" you need to set the argument "skip ready" to "True".
Example:
Click Element xpath=//a[@href="#/motor"]
Becomes
Click Element xpath=//a[@href="#/motor"] True